Methods and apparatuses for dental images

ABSTRACT

Described herein are methods and apparatuses to obtain an image, or a set of images, of a patient&#39;s teeth from one or more predetermined viewing angles. These methods and apparatuses may include the use an overlay comprising an outline of teeth for each predetermined viewing angle. The overlay may be used for automatically capturing, focusing and/or illuminating the teeth. Also described herein are methods and apparatuses for using a series of images of the patient&#39;s teeth including a set of predetermined views to determine if a patient is a candidate for an orthopedic procedure.

CLAIM OF PRIORITY

This patent application is a continuation of U.S. patent applicationSer. No. 17/543,637, filed Dec. 6, 2021, titled “METHODS AND APPARATUSESFOR DENTAL IMAGES,” now U.S. Patent Application Publication No.2022/0087778, which is a continuation of U.S. patent application Ser.No. 16/952,072, filed Nov. 18, 2020, titled “METHODS AND APPARATUSES FORDENTAL IMAGES,” which is a continuation of U.S. patent application Ser.No. 16/827,594, filed Mar. 23, 2020, titled “METHODS AND APPARATUSES FORDENTAL IMAGES,” now U.S. Pat. No. 10,932,885, which is a continuation ofU.S. patent application Ser. No. 15/803,718, filed Nov. 3, 2017, titled“METHODS AND APPARATUSES FOR DENTAL IMAGES,” now U.S. Pat. No.10,595,966, which claims priority to U.S. Provisional Patent ApplicationNo. 62/417,985, filed Nov. 4, 2016, and titled “METHODS AND APPARATUSESFOR DENTAL IMAGES,” each of which is herein incorporated by reference inits entirety.

INCORPORATION BY REFERENCE

All publications and patent applications mentioned in this specificationare herein incorporated by reference in their entirety to the sameextent as if each individual publication or patent application wasspecifically and individually indicated to be incorporated by reference.

FIELD

This disclosure relates generally to methods and apparatuses for theanalysis of dental images, including methods and apparatuses forcapturing dental images and methods and apparatuses for remotelypre-screening a patient for an orthodontic treatment.

BACKGROUND

In dental and/or orthodontic treatment, a set of 2D facial and dentalphotos is often taken. Traditional dental photography uses a camera, forexample, a digital single-lens reflex (SLR) camera with a lens with afocal length of 90-100 mm and circular flash as shown in FIGS. 1A-1C.However, the SLR camera is expensive and it may be difficult to alignthe teeth of a patient to the SLR camera. The process may also beuncomfortable to the patient. Some doctors and dental technicians mayinstead attempt to capture dental photos using a mobile phone. However,mobile phones typically have a wide angle camera lens. If a camera of amobile phone is held sufficiently close to the teeth of the patient toprovide an image of the teeth having a high level of detail, dentalphotos may be blurry and have optical distortion. If the camera of themobile phone is too far to the teeth, dental photos cannot meet theorthodontic standards. In general, commonly available cameras, andparticularly mobile phone cameras, may be faster and easier to use.

Thus, there is a need for new and useful methods and apparatuses forobtaining high quality dental images.

SUMMARY OF THE DISCLOSURE

Described herein are methods, apparatuses (including devices andsystems, such as non-transitory, computer-readable storage media andsystems including these) for capturing dental images and for using thesedental images to determine if a patient is a candidate for anorthodontic procedure. In general, the methods and apparatuses describedherein may obtain an image of a patient's teeth for therapeutic use,which may include viewing the patient's teeth, for example, on a screenof a mobile telecommunications device (such as a mobile phone or otherhand-held personal computing device, e.g., smartwatch, pad, laptop,etc.).

Any of the methods described herein may include guiding or assisting intaking a predetermined set of images of the patient's teeth fromspecified viewing angles. The specified viewing angles (views) may beused to manually or automatically determine if a patient is a candidatefor a particular orthodontic treatment. Often, it may be necessary orhelpful that the views are taken with the proper resolution and focus,particularly when automatically analyzing the images later. Thus, in anyof the methods and apparatuses described herein, an overlay may be shownover a camera image while preparing to take a picture of the teeth. Theoverlay may guide the user (e.g., dental technician, including dentist,orthodontist, dental assistant, nurse, etc.) in taking the images.Further, the overlay may be used to aid in focusing and illuminating theteeth during collection of the images. For example, the methodsdescribed herein may include displaying, on a screen (e.g., of a mobilecommunications device such as a smartphone, etc.), an overlay comprisingan outline of teeth in a predetermined view. The overlay may bedisplayed atop the view from the camera, which may showing a real-timedisplay of from the camera. As the camera is used to image the patient'steeth, the mobile communications device may be moved so that the overlayapproximately matches the patient's teeth in the view of the patient'steeth. The method can further comprise capturing an image of the view ofthe patient's teeth.

For example, a method of obtaining an image of a patient's teeth fortherapeutic use is described herein. The method can comprise viewing thepatient's teeth, for example, on a screen of a mobile telecommunicationsdevice having a camera (e.g., smartphone, mobile phone, etc.). Themethod can further comprise displaying, on the screen, an overlaycomprising an outline of teeth in a predetermined view, wherein theoverlay is displayed atop the view of the patient's teeth. The methodcan comprise moving the mobile telecommunications device relative to thepatient's teeth and triggering an indicator when the overlayapproximately matches with the patient's teeth. The method can comprisecapturing an image of the view of the patient's teeth when the indicatoris triggered.

A method to obtain and an image of a patient's teeth for therapeutic usemay include viewing, on a screen of a mobile telecommunications device,the patient's teeth. The method can further comprise displaying, on thescreen, an overlay comprising a cropping frame and an outline of teethin one of an anterior view, a buccal view an upper jaw view, or a lowerjaw view, wherein the overlay is displayed atop the view of thepatient's teeth. The method can further comprise moving the mobiletelecommunications device so that the overlay approximately matches thepatient's teeth in the view of the patient's teeth. The method cancomprise capturing an image of the view of the patient's teeth. Themethod can further comprise reviewing the captured image on the mobiletelecommunications device and indicating on the screen of the mobiletelecommunications device if the captured image is out of focus. Themethod can further comprise automatically cropping the captured image asindicated by the cropping frame.

For example, the overlay can comprise a generic overlay in someembodiments. For another example, the overlay can comprise apatient-specific overlay derived from the patient's teeth in some otherembodiments.

For example, the method can further comprise automatically triggering anindicator when the overlay approximately matches with the patient'steeth. For example, the method can further comprise triggering theindicator when the overlay approximately matches with the patient'steeth comprises estimating an indicator of the distance between an edgeof the patient's teeth in the view of the patient's teeth and theoutline of teeth in the overlay. For another example, the method canfurther comprise triggering the indicator when the overlay approximatelymatches with the patient's teeth comprises estimating an indicator ofthe distance between an edge of the patient's teeth at two or moreregions and the outline of teeth and comparing that indicator to athreshold value. For example, the indicator can be a visual indicator,such as a change of color. In some variations, the indicator can beother forms of indicators, such as a voice indicator.

The method can further comprise automatically capturing an image of thepatient's teeth when the overlay approximately matches with thepatient's teeth.

The method can further comprise checking the image quality of thecaptured image and displaying on the screen if the image quality isbelow a threshold for image quality.

The method can further comprise cropping the captured image based on acropping outline displayed as part of the overlay. Cropping may bemanual or automatic.

Any of these methods can further comprise evaluating the captured imagefor medical treatment by using the image. For example, the method canfurther comprise transmitting the captured image to a remote server.

The predetermined view can comprise an anterior view, a buccal view anupper jaw view, or a lower jaw view. The predetermined view can comprisea set of dental images according to the orthodontic standards. Forexample, the method can further comprise repeating the steps of viewing,displaying, moving and capturing to capture anterior, buccal, upper jawand lower jaw images of the patient's teeth.

The method can further comprise imaging a patient's identification usingthe mobile telecommunications device and automatically populating a formwith user identification information based on the imaged identification.

Any of these methods can further comprise displaying instructions aboutpositioning the patient's teeth on the screen of the mobiletelecommunications device prior to displaying the overlay.

Also described herein are apparatuses adapted to perform any of themethods described herein, including in particular software, firmware,and/or hardware adapted to perform one or more of these methods.Specifically, described herein are non-transitory, computer-readablestorage media storing a set of instructions capable of being executed bya processor (e.g., of a mobile telecommunications device), that, whenexecuted by the processor, causes the processor to display real-timeimages of the patient's teeth on a screen of the mobiletelecommunications device, display an overlay comprising an outline ofteeth in a predetermined view atop the images of the patient's teeth,and enable capturing of an image of the patient's teeth.

For example, described herein are non-transitory, computer-readablestorage media storing a set of instructions capable of being executed bya processor of a mobile telecommunications device, that, when executedby the processor, causes the processor to display real-time images ofthe patient's teeth on a screen of the mobile telecommunications device,display an overlay comprising an outline of teeth in a predeterminedview atop the images of the patient's teeth, trigger an indicator whenthe overlay approximately matches with the patient's teeth, and enablecapturing of an image of the patient's teeth when the indicator istriggered.

Also described herein are non-transitory, computer-readable storagemedium storing a set of instructions capable of being executed by aprocessor of a mobile telecommunications device, that, when executed bythe processor, causes the processor to display real-time images of thepatient's teeth on a screen of the mobile telecommunications device anddisplay an overlay comprising a cropping frame and an outline of teethin one of an anterior view, a buccal view an upper jaw view, or a lowerjaw view, wherein the overlay is displayed atop the images of thepatient's teeth, and enable capturing of an image of the patient'steeth. The non-transitory, computer-readable storage medium, wherein theset of instructions, when executed by the processor, can further causethe processor to review the captured image and indicate on the screen ifthe captured image is out of focus and automatically crop the capturedimage as indicated by the cropping frame.

The set of instructions, when executed by the processor, can furthercause the processor to display a generic overlay. For another example,set of instructions can cause the processor to display apatient-specific overlay derived from the patient's teeth.

The set of instructions, when executed by the processor, can furthercause the processor to automatically trigger an indicator when theoverlay approximately matches with the patient's teeth. For example, theset of instructions can further cause the processor to estimate anindicator of the distance between an edge of the patient's teeth in theview of the patient's teeth and to trigger the indicator when theoutline of teeth in the overlay is less than or equal to a thresholdvalue. The set of instructions, when executed by the processor, canfurther cause the processor to estimate an indicator of the distancebetween an edge of the patient's teeth at two or more regions in theview of the patient's teeth and to trigger the indicator when theoutline of teeth in the overlay is less than or equal to a thresholdvalue. The set of instructions can cause the processor to trigger theindicator by displaying a visual indicator on the screen. Anyappropriate visual indicator may be displayed, including a color,intensity (e.g., changing the color and/or intensity of the outline ofthe teeth overlay, cropping window, etc.), a textual/characterindicator, or some combination thereof. Alternatively or additionallythe indicator may be audible (beeping, tonal, etc.) and/or tactile (avibration, buzzing, etc.).

The set of instructions, when executed by the processor, can furthercause the processor to check the image quality of the captured image anddisplaying on the screen if the image quality is below a threshold forimage quality. The quality may automatically determine focus, lighting(dark/light), etc. of the image and may alert the user and/orautomatically reject or accept the image. The apparatus may furtherprocess the image (e.g., sharpen, lighten/darken, etc., includingcropping). For example, the non-transitory, computer-readable storagemedium, wherein the set of instructions, when executed by the processor,can further cause the processor to automatically crop the captured imagebased on a cropping outline displayed as part of the overlay.

The set of instructions, when executed by the processor, can furthercause the processor to transmit the captured image to a remote server.Transmission may be automatic or manual.

The apparatus (e.g., including the non-transient set of instructions)can further cause the processor to display an overlay comprising anoutline of teeth in a predetermined view such as an anterior view, abuccal view an upper jaw view, or a lower jaw view. The apparatus may beconfigured to take a full or partial set of views. For example, the setof instructions, when executed by the processor, can further cause theprocessor to repeat the steps of viewing, displaying, moving andcapturing to capture anterior, buccal, upper jaw and lower jaw images ofthe patient's teeth.

In addition to taking one or more views (e.g., anterior, buccal, upperjaw and lower jaw) the apparatuses described herein may be configured toautomatically determine patient-specific information on the identity andother patient characteristics, and to associate this information withthe images taken. For example, the set of instructions can cause theprocessor to capture an image of a patient's identification (e.g.,driver's license) using the mobile telecommunications device andautomatically populate a form with user identification information basedon the imaged identification.

In any of these apparatuses the set of instructions can further causethe processor to display instructions on positioning the patient's teethon the screen of the mobile telecommunications device prior todisplaying the overlay.

Any of the methods described herein may be methods to obtain a series ofimages of a patient's teeth. These methods may include: displaying, on ascreen of a mobile telecommunications device having a camera, areal-time image from the camera; and guiding a user in taking a seriesof predetermined views of the patient's teeth by sequentially, for eachpredetermined view: displaying, superimposed over the real-time image onthe screen, an overlay comprising an outline of teeth in a predeterminedview from the plurality of predetermined views; and triggering thecapture of an image of the patient's teeth when the overlayapproximately matches with the image of the patient's teeth in thedisplay on the screen.

For example, a method to obtain a series of images of a patient's teethmay include: displaying, on a screen of a mobile telecommunicationsdevice having a camera, the patient's teeth; and guiding a user intaking a series of predetermined views of the patient's teeth bysequentially, for each predetermined view: displaying, on the screen, anoverlay comprising an outline of teeth in a predetermined view, whereinthe overlay is displayed atop the view of the patient's teeth;automatically adjusting the camera to focus within a region of thescreen that is within the overlay; automatically adjusting a lightemitted by the camera based on a level of light of within the region ofthe screen that is within the overlay; and triggering the capture of animage of the patient's teeth when the overlay approximately matches withthe patient's teeth in the display of the patient's teeth, wherein thepredetermined views include at least one each of: an anterior view, abuccal view, an upper jaw view, and a lower jaw view.

A series of images of a patient's teeth may include a set, collection,or grouping. A series may be organized or ordered, e.g., in a predefinedorder based on the predefined views (e.g., viewing angles). The seriesof images may be collected or linked to together, and may includeidentifying information, including information identifying thecorresponding viewing angle (e.g., anterior view, buccal view, an upperjaw view, a lower jaw view, etc.). In any of these variations additionalinformation may be included, such as the user and/or patient's chiefdental concern (e.g., crowding, spacing, smile width, arch width, smileline, horizontal overjet, vertical overbite, cross bite, biterelationship, etc.). In general, the set of images may refer to a seriesof predetermined views. The predetermined views may refer topredetermined viewing angles for visualizing the teeth. Viewing anglesmay refer to the view of the upper and/or lower dental arch, and mayinclude, for example: anterior (e.g., upper and lower anterior,typically with a closed bite), anterior open bite, right buccal(typically with a closed bite), right buccal open bite, left buccal(typically with a closed bite), left buccal open bite, upper jaw (e.g.,viewed from occlusal surface), and lower jaw (e.g., viewed from anocclusal surface). The predetermined views may also include views thatinclude the entire head, e.g., a head profile, a facial view (typicallywith the mouth closed, unsmiling), as well as a facial view with thepatient smiling. A dental mirror may be used to take the upper and lowerjaw images. The systems and methods described herein may automaticallydetermine if a mirror is used, and may orient the image accordingly.

Any of the methods and apparatuses described herein may guide a user intaking a series. The method or apparatus may provide audible and/orvisual instructions to the user. In particular, as mentioned above, anyof these apparatuses may include an overlay on the display (screen) ofthe mobile telecommunications device showing an outline that may bematched to guide the user in taking the image(s). The overlay may beshown as an outline in a solid and/or semi-transparent color. An overlaymay be shown for each predetermined view. The user may observe thescreen and, once the image shows the patient's anatomy approximatelymatching within the overlay, the image may be captured. Image capturemay be manual (e.g., manually triggered for capture by the useractivating a control, such as pushing a button to take the image) and/orautomatic (e.g., detected by the system and automatically triggered totake the image when the overlay is matched with the correspondingpatient anatomy). In general, capturing or triggering the capture of theimage of the patient's teeth (and/or the patient's head) when theoverlay approximately matches with the image of the patient's teeth inthe display may refer to automatic capturing/automatic triggering,semi-automatic capturing/semi-automatic triggering, or manualcapturing/manual triggering. Automatic triggering (e.g., automaticcapturing) may refer to automatic capture of the image, e.g., taking oneor more images when the patient's anatomy (e.g., teeth) show on thescreen matches the overlay on the screen. Semi-automatic triggering(e.g., semi-automatic capturing) may refer to producing a signal, suchas an audible sound and/or visual indicator (e.g., flashing, colorchange, etc.) when the patient's anatomy (e.g., teeth) shown on thescreen matches the overlay on the screen. Manual triggering (e.g.,manual capturing) may refer to the user manually taking the image, e.g.,taking one or more images when the patient's anatomy (e.g., teeth) isshown on the screen to match the overlay.

As described in greater detail herein, automatic or semi-automatictriggering (e.g., automatic or semi-automatic capture of images) may beaccomplished by a variety of well-known image processing techniques. Forexample, detection of a match between the patient's anatomy (e.g.,teeth) and the overlay on a screen may be achieved by edge detection;the edge of the patient's teeth may be compared to the overly region andif two or more regions (e.g., two opposite regions, etc.) are within adefined distance (e.g., +/−1 mm, +/−2 mm, +/−3 mm, +/−4 mm, +/−5 mm,+/−6 mm, +/−7 mm, +/−8 mm, +/−10 mm, etc. or +/−a corresponding numberof pixels for the image, +/−a percentage, such as 1%, 2%, 3%, 5%, 7%,10%, etc. of the screen diameter, etc.). The automatic detection ofmatch may be determined by machine learning, e.g., training a machine torecognize matching of the patient anatomy (e.g., teeth) within theoverlay with an acceptable percentage of match.

Any of these methods may include displaying on a screen of the mobiletelecommunications device, images, and particularly real-time images,from the cameral of the mobile telecommunications device. Real-time maybe refer to the current, or approximately concurrent, display of imagesdetected by the camera on a screen or screens, e.g., of the mobiletelecommunications device.

In general, the overlay may also be used to improve the image quality.Of the image(s) being taken. For example, any of these methods andapparatuses may automatically focus the imaging only within the regiondefined by the overlay. For example, any of these methods andapparatuses may disable or modify the autofocusing of the cameral of themobile telecommunications device (e.g., mobile phone) and may autofocuson just the region within the overlay, or a sub-region within theoverlay (e.g., on the anterior teeth, the incisor, canine, bicuspid,molars, etc.).

The overly may also control the illumination (e.g., lighting) of theimages based on the region within all or a portion of the overlay. Forexample, the apparatus or method may detect and adjust the light levelbased on the light level within the overlay or a sub-region within theoverlay (e.g., on the incisors, canines, bicuspids, molars, etc.). Theillumination may generally be provided by the mobile telecommunicationsdevice, which may include a flash or LED light source that can beadjusted for continuous and/or discrete illumination.

In any of the methods and apparatuses described herein, the images takenfor particular views (e.g., anterior, anterior open bite, right buccal,right buccal open bite, left buccal, left buccal open bite, upper jaw,and lower jaw, etc.) may be labeled with the corresponding view, eithermanually or automatically. Further, the view may be detected andidentified by the method or apparatus. In variations in which theoverlay for a particular view is provided before taking the image, theview shown in the overlay may determine the label for the resultingimage. As mentioned herein, in some variations, automatic detection ofthe nearest view may be performed on the imaging, and the view (viewingangle) may be detected automatically. Additionally or alternatively,mirror images, may be detected or identified, and the resulting imagesflipped/rotated, and/or labeled to indicate that a dental mirror wasused to take the image.

In any of the methods and apparatuses described herein, the overlaydisplayed over an image on the screen of the mobile telecommunicationdevice may be selected automatically, e.g., by identifying the closestmatch to one of the predetermined viewing angles. The overly having thecloset match may then be used to take an image for the set of images.Alternatively or additionally, the overlay may be provided first, andthe user may then move the camera portion of the mobiletelecommunications device to fit the patient's anatomy into the overlay.

As mentioned, any of the images for predetermined viewing angles (views)described herein may be taken with the use of a cheek retainer. Anapparatus instruction the user to take the images may include written,pictorial (visual) and/or audible instruction on the use of the cheekretainer. Any of the methods and apparatuses described herein mayautomatically detect a cheek retainer; this may aid in automaticlabeling and interpretation of the resulting image(s). In somevariations the apparatus and/or method may detect one or more markers onthe cheek retainer and use this information identify a view, to identifya match between an image and an overlay, etc.

Also described herein are methods and apparatuses for remotelypre-screening a patient for an orthodontic treatment. Any orthodontictreatment may be attempted, particularly orthodontic treatments foraligning the patient's teeth. Typically such methods may include takinga series of predetermined views of the patient's teeth, and optionallycollecting information about the patient, such as one or more chiefdental concerns (e.g., a chief patient dental concern such as, e.g.,crowding, spacing, smile width, arch width, smile line, horizontaloverjet, vertical overbite, cross bite, bite relationship, etc.). Thisadditional information may be linked to the series of images, and may beused, along with the series of images, to determine if a patient is, oris not, a good candidate for an orthodontic treatment.

For example, described herein are methods for remotely pre-screening apatient for an orthodontic treatment, the method comprising: guiding auser, with a mobile telecommunications device having a camera, to take aseries of images of the patient's teeth in a plurality of predeterminedviews; transmitting the series of images from the mobiletelecommunications device to a remote location to determine if thepatient is, or is not, a candidate for the orthodontic treatment basedon the series of images; and displaying, on a screen of the mobiletelecommunications device, an indicator that the patient is, or is not,a candidate for the orthodontic treatment.

In any of these methods, guiding may refer to sequentially, for eachpredetermined view, displaying, on the screen, an overlay comprising anoutline of teeth in one of the predetermined views from the plurality ofpredetermined views, wherein the overlay is displayed atop an image ofthe patient's teeth. The overlay may be provided first, or the overlaymay be selected from the set of overlay viewing angles that best matchesthe current view being imaged by the mobile telecommunications device.Alternatively, the predetermined views may be presented in a fixedorder.

As mentioned, guiding the user may include, for each predetermined view,capturing an image of the patient's teeth when the image of thepatient's teeth approximately matches an overlay corresponding to apredetermined view. Capturing may be manual, automatic orsemi-automatic, as discussed above. For example, any of these methodsmay include automatically determining when the image of the patient'steeth approximately matches the overlay by detecting an edge of thepatient's teeth and comparing the detected edge to the overlay.

Any of these methods and apparatuses may include automatically adjustingthe camera to focus the camera within a region of the screen that iswithin the overlay. This region may include all of the region within theoverlay, or a sub-set (e.g., corresponding to the anterior teeth, theposterior teeth, etc.

Any of these methods and apparatuses may include selecting the overlaybased on one or more images of the patient's teeth. This may includeselecting the overlay corresponding to the particular viewing angle, asmentioned above, and/or it may include customizing the overlay based onthe patient's specific anatomy. For example the overlay maybe selectedto match the shape, size and arrangement of the patient's dentition.

Any of these methods and apparatuses may include automatically adjustingthe light emitted by the camera based on a level of light of within aregion of the screen that is within the overlay. The light may becontinuous or intermittent (e.g., flash). Thus, the apparatus or methodmay first disable the default light sensing for the mobiletelecommunications device, and may instead use the region (or asub-section of the region) within the overlay to set the light level foradjusting the flash/applied light from the mobile telecommunicationsdevice.

As mentioned, any of these methods and apparatuses may be configured tocapture the image of the patient's teeth when the image of the patient'steeth approximately matches the overlay corresponding to thepredetermined view, e.g., by automatically capturing the image.Similarly, any of these methods and apparatuses may capture the image ofthe patient's teeth when the image of the patient's teeth approximatelymatches the overlay corresponding to the predetermined view bysemi-automatically capturing, e.g., triggering a visual, audible, orvisual and audible indicator that permits the user to take the image. Insome variations a plurality of images may be taken and averaged or usedto select the best image.

Transmitting the series of images from the mobile telecommunicationsdevice to the remote location may generally include receiving, in themobile telecommunications device, an indication that patient is, or isnot, a candidate within a fixed period of time (e.g., 10 minutes, 15minutes, 20 minutes, etc.) from transmitting the series of images. Ingeneral, the initial decision that a patient is a good candidate for theorthodontic treatment may use the set of images transmitted, and mayalso include the chief concern. The decision may be made at the remotelocation (e.g., a remote server, etc.) either manually or automatically.Automatic decisions may be based on the amount of movement required toposition the teeth in order to or address the chief concern and/or astandard of orthodontic positioning. The methods and apparatusesdescribe herein may provide images with sufficient clarity so thatindividual tooth positions may be determined relative to the dental archand used to at least roughly approximate the complexity of anorthodontic procedure. Cases in which the amount and/or type of movementis complex may be indicated as not candidates. Cases in which the amountand/or type of movement is not complex may be indicated as candidates.Complex dental movements may include movements of greater than a minimumthreshold (e.g., greater than 3 mm distal/proximal movement, greaterthan 4 mm distal/proximal movement, 5 mm distal/proximal movement,greater than 6 mm distal/proximal movement, greater than 7 mmdistal/proximal movement, etc.), and/or rotation of greater than aminimum threshold (e.g., greater than 5, 10, 15, 20, 25, 30, 35, 45,etc., degrees), and/or extruding of one or more teeth greater than aminimum threshold (e.g., greater than 0.5 mm, 1 mm, 2 mm, 3 mm, 4 mm,etc.).

As mentioned, in general, any of these methods and apparatuses mayinclude automatically identifying each image of the series of images toindicate which view of the plurality of views each image includes (e.g.,anterior, anterior open bite, right buccal, right buccal open bite, leftbuccal, left buccal open bite, upper jaw, lower jaw, etc.). Any of thesemethods may also include automatically determining if one or more of theseries of images was taken using a mirror. For example, the image may beautomatically examined to identify reflections (e.g., a plane ormirror), and/or to determine if the orientation of the teeth within theimage are reversed (mirrored) in the image. Mirrored images may bereversed for display as a non-mirrored image. Alternatively oradditionally, duplicate mirrored regions may be cropped from the image.

Any of the methods and apparatuses described herein may includereceiving, in the mobile telecommunications device, an indication of thepatient's chief dental concern and aggregating the patient's chiefdental concern with the series of images. Transmitting the series ofimages may comprise transmitting the aggregated series of images and thepatient's chief dental concern.

As mentioned, any of the methods and apparatuses described herein may beconfigured to include instructing the user to retract the patient'scheek with a cheek retractor. A marker on the cheek retractor may beused to automatically identify the image to indicate which view of theplurality of views it includes based on the identified cheek retractor.

Although the terms “user” and “patient” are used separately herein, theuser may be the patient. For example, the person taking the images usingthe methods and apparatuses described herein may be the patient. Thus,in any of these methods, the user may be the patient. Alternatively, aseparate use (e.g., dentist, orthodontist, dental technician, dentalassistant, etc.) may act as the user, taking the images as describedherein on a patient.

A method for remotely pre-screening a patient for an orthodontictreatment may include: guiding a user, with a mobile telecommunicationsdevice having a camera, to take a series of images of the patient'steeth in a plurality of predetermined views by sequentially, for eachpredetermined view: displaying, on the screen, an overlay comprising anoutline of teeth in one of the predetermined views from the plurality ofpredetermined views, wherein the overlay is displayed atop an image ofthe patient's teeth; and capturing the image of the patient's teeth whenthe overlay approximately matches the patient's teeth in the view of thepatient's teeth; transmitting the series of images to a remote locationto determine if the patient is a candidate for the orthodontic treatmentbased on the series of images; and displaying, on the screen of themobile telecommunications device, an indicator that the patient is, oris not, a candidate for the orthodontic treatment.

A method for remotely pre-screening a patient for an orthodontictreatment may include: guiding a user, with a mobile telecommunicationsdevice having a camera, to take a series of images of the patient'steeth from a plurality of predetermined views by sequentiallydisplaying, on a screen of the mobile telecommunications device, anoverlay comprising an outline of teeth in each of the predeterminedviews; receiving, in the mobile telecommunications device, an indicationof the patient's chief dental concern; aggregating, in the mobiletelecommunications device, the series of images and the chief dentalconcern; transmitting the aggregated series of images and the chiefdental concern to a remote location to determine if the patient is acandidate for the orthodontic treatment based on the series of images;and displaying, on the screen of the mobile telecommunications device,an indicator that the patient is, or is not, a candidate for theorthodontic treatment.

Any of the methods (and method steps) described herein may be performedby an apparatus configured to perform the method(s). For example,described herein are systems for remotely pre-screening a patient for anorthodontic treatment. A system for remotely pre-screening a patient(e.g., remote to the patient) may include: a non-transitory,computer-readable storage medium storing a set of instructions capableof being executed by a processor of a mobile telecommunications devicehaving a camera, that, when executed by the processor, causes theprocessor to: guide a user to take a series of images of the patient'steeth in a plurality of predetermined views with the camera; transmitthe series of images from the mobile telecommunications device to aremote location to determine if the patient is a candidate for theorthodontic treatment based on the series of images; and display, on ascreen of the mobile telecommunications device, an indicator that thepatient is, or is not, a candidate for the orthodontic treatment. Thenon-transitory, computer readable storage medium may cause the processorto guide the user to take the series of images of the patient's teethby: displaying, on the screen of the mobile telecommunications device,an image from the camera and an overlay comprising an outline of teethin one of the predetermined views from the plurality of predeterminedviews, wherein the overlay is displayed atop the image from the camera.The non-transitory, computer readable storage medium may cause theprocessor to guide the user to take the series of images of thepatient's teeth by automatically adjusting the camera to focus thecamera within a region of the screen that is within the overlay. Thenon-transitory, computer readable storage medium may causes theprocessor to guide the user to take the series of images of thepatient's teeth by selecting the overlay based on one or more images ofthe patient's teeth. The non-transitory, computer readable storagemedium may cause the processor to guide the user to take the series ofimages of the patient's teeth by automatically adjusting a light emittedby the camera based on a level of light of within a region of the screenthat is within the overlay.

The non-transitory, computer readable storage medium may cause theprocessor to guide the user to take the series of images of thepatient's teeth by: indicating when an overlay comprising an outline ofteeth in one of the predetermined views from the plurality ofpredetermined views aligns with a view of the patient's teeth from thecamera, wherein the overlay is displayed atop the view from the camera.The non-transitory, computer readable storage medium may cause theprocessor to guide the user to take the series of images of thepatient's teeth by: automatically taking an image of the patient's teethwhen an overlay comprising an outline of teeth in one of thepredetermined views from the plurality of predetermined views alignswith a view of the patient's teeth from the camera, wherein the overlayis displayed atop the view from the camera.

The non-transitory, computer readable storage medium may cause theprocessor to guide the user to take the series of images of thepatient's teeth comprises guiding the user to take at least one each of:an anterior view, a buccal view, an upper jaw view, and a lower jawview.

Any of these systems may be configured so that the non-transitory,computer readable storage medium further causes the processor toreceive, in the mobile telecommunications device, an indication thatpatient is, or is not, a candidate within 15 minutes of transmitting theseries of images. The non-transitory, computer readable storage mediummay cause the processor to automatically identify each image of theseries of images to indicate which view of the plurality of views eachimage includes. The non-transitory, computer readable storage medium maycause the processor to automatically determine if one or more of theseries of image was taken using a mirror.

The non-transitory, computer readable storage medium may further causethe processor to receive, in the mobile telecommunications device, anindication of the patient's chief dental concern and aggregating thepatient's chief dental concern with the series of images, furtherwherein the on-transitory, computer readable storage medium may beconfigured to transmit the series of images as the aggregated series ofimages and the patient's chief dental concern.

The non-transitory, computer readable storage medium may further causethe processor to instruct the user to retract the patient's cheek with acheek retractor, and/or to identify a marker on the cheek retractor andto mark an image from the plurality of predetermined views to indicatewhich view of the plurality of views it includes based on the identifiedcheek retractor.

Any of the systems described herein may include a remote processorconfigured to receive the transmitted series of images and to transmitan indicator that the patient is, or is not, a candidate for theorthodontic treatment based on the series of images back to thenon-transitory, computer-readable storage medium.

For example, a system for remotely pre-screening a patient for anorthodontic treatment may include a non-transitory, computer-readablestorage medium storing a set of instructions capable of being executedby a processor of a mobile telecommunications device having a camera,that, when executed by the processor, causes the processor to: guide auser to take a series of images of the patient's teeth in a plurality ofpredetermined views by sequentially, for each predetermined view:displaying, on a screen of the mobile telecommunications device, animage from the camera and an overlay comprising an outline of teeth inone of the predetermined views from the plurality of predeterminedviews, wherein the overlay is displayed atop the image from the camera;and capturing the image when the overlay approximately matches thepatient's teeth on the screen; and transmit the series of images to aremote location.

A system for remotely pre-screening a patient for an orthodontictreatment may include: a non-transitory, computer-readable storagemedium storing a set of instructions capable of being executed by aprocessor of a mobile telecommunications device having a camera, that,when executed by the processor, causes the processor to: guide a user totake a series of images of the patient's teeth in a plurality ofpredetermined views by sequentially, for each predetermined view:displaying, on a screen of the mobile telecommunications device, animage from the camera and an overlay comprising an outline of teeth inone of the predetermined views from the plurality of predeterminedviews, wherein the overlay is displayed atop the image from the camera;and capturing the image when the overlay approximately matches thepatient's teeth on the screen; and transmit the series of images to aremote location; and a remote processor configured to receive thetransmitted series of images and to transmit an indicator that thepatient is, or is not, a candidate for the orthodontic treatment basedon the series of images back to the non-transitory, computer-readablestorage medium.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe claims that follow. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings of which:

FIGS. 1A-1C illustrate one example of traditional dental photographyusing a digital SLR camera. FIG. 1A shows an example of a traditionalSLR camera taking an anterior open bite view of a patient's teeth. FIG.1B shows an anterior view (closed bit) taken with the traditional SLRcamera, using a pair of cheek retractors that the patient holds onto, asshown in FIG. 1C.

FIG. 2A illustrates an example of a screen of an apparatus (such as asystem configured to perform the method as described herein) includingan overlay comprising an outline of teeth in a predetermined view, shownas an anterior view of teeth, on a screen of a mobile telecommunicationsdevice.

FIG. 2B illustrates another example of an example of the screen of theapparatus, similar to that shown in FIG. 2B, showing the overlay andanother anterior view of teeth. For purposes of illustration the modelis shown as a dental model (e.g., positive cast) of a patient'sdentition; the teeth may be directly imaged, either with or without aretractor.

FIG. 2C shows an example of a screen of an apparatus configured to takeimages (e.g., photographs) of a patient's teeth in a left buccal openbite configuration, including an overlay on the screen which shows whatteeth should be visible in the view.

FIG. 3A illustrates an example of a patient-specific overlay accordingto another embodiment of the disclosure.

FIG. 3B illustrates an example of an indicator which is triggered whenthe overlay approximately matches with the patient's teeth. The triggermay be visual (including changing the color of the overlay, displayingor flashing an image/icon/color/symbol on the screen, etc.) and/oraudible (emitting a ping, tone, etc.), and/or tactile (e.g., vibrating),etc.

FIGS. 4A-4H illustrate 8 specific overlay images for the required typesof photos according to the orthodontic standard, including an anteriorview in FIG. 4A, an another anterior view in FIG. 4B, an upper jaw viewin FIG. 4C, a lower jaw view in FIG. 4D, a left buccal view in FIG. 4E,an another left buccal view in FIG. 4F, a right buccal view in FIG. 4G,and an another right buccal view in FIG. 4H.

FIG. 5A is an example of a user interface to enable a user to select oneof a plurality of overlays for a plurality of dental images in aplurality of predetermined views.

FIG. 5B is an example of on-screen message to guide a user to takedental images.

FIG. 5C is an example of a user interface to take a profile image of apatient.

FIG. 5D is an example of a user interface to take a facial image of thepatient.

FIG. 6A is an example screenshot which indicates the captured image isout of focus after reviewing the image quality.

FIG. 6B is an example of a poor quality image of the teeth.

FIG. 7A shows an example overlay comprising a cropping frame and anoutline of teeth in an upper jaw view.

FIG. 7B shows another example overlay comprising a cropping frame and anoutline of teeth in anterior view.

FIG. 7C shows an example of an image captured using the overlay of FIG.7B.

FIG. 8A is an example of a block diagram of a method to obtain an imageof teeth of a patient.

FIG. 8B is an example of a method of remotely pre-screening a patientfor an orthodontic treatment using the method of FIG. 8A.

FIG. 9 is an example of a set of facial and dental images of a patientcaptured by using a digital SLR camera.

FIG. 10 is an example of a set of facial and dental images of a patientcaptured by using a mobile telecommunications device through the methoddisclosed herein.

FIGS. 11A-11H are examples of a set of dental images captured by usingthe mobile telecommunications device through the method disclosedherein.

FIG. 12A illustrates a method of focusing within a region of an overlay(shown by the dot on the upper incisor) rather than defaulting to theautofocus of the camera for the mobile telecommunications device.

FIGS. 12B and 12C illustrate a comparison between using the defaultautofocus (FIG. 12B) of a mobile telecommunications device, and thetargeted focus (e.g., as shown in FIG. 12A) with the systems and methodsdescribed herein, providing a shaper overall focus on the teeth. In theline drawings shown (adapted from photographs) the focus is illustratedby the relative darkness of the lines. The focus may be represented bythe darkness of the lines; in FIG. 12B the focus is on the lips andgingiva, while in FIG. 12C, the focus is on the teeth, which will appearmore in-focus compared to FIG. 12B.

FIGS. 13A and 13B show a first example of a comparison between using thedefault focus and lighting (flash), as shown in FIG. 13A, compared withusing a region within the overlay (overlay not shown in FIGS. 13A and13B) to set the focus and lighting level. In FIGS. 13A and 13B, the linedrawings are adapted from photographs showing images taken without andwith, respectively, the lighting being adjusted based on the overlayregion.

FIGS. 13C and 13D illustrate another example of a method of using aregion within the overlay (overlay not shown in FIGS. 13A and 13B) toset the focus and lighting level.

FIG. 14A shows an example of a patient manually retracting their cheek.FIG. 14B shows a patient retracting their cheeks without the aid of afiner or retractor.

FIG. 14C is an example of a cheek retractor that may be used to aid inretracting the cheeks. In FIG. 14C the retractor includes one or moremarkers that may be used to identify the position of the retractor (andtherefore the patient's teeth) relative to the view.

FIG. 15 is an example of a view of a patient using a retractor. In FIG.15, the patient is positioned a distance from the camera, and themethods and apparatuses described herein may indicate that the camerashould be moved closer.

FIG. 16 shows a patient for which facial recognition may be used toprovide information about the images.

DETAILED DESCRIPTION

The following description of the various embodiments of the invention isnot intended to limit the invention to these embodiments, but rather toenable any person skilled in the art to make and use this invention.

Described herein are methods and apparatuses (including devices,systems, non-transitory, computer-readable storage medium storinginstructions capable of being executed by a processor, etc.) forcapturing high quality dental images, including obtaining an image orset of images at predetermined positions of a patient's teeth and/orface for therapeutic use. In particular, described herein are methodsand apparatuses for remotely pre-screening a patient for an orthodontictreatment that includes taking a defined set of images of the patient'steeth at known angles, and transmitting them as a set for remoteanalysis. The images must be at the predetermined angles and must besufficiently well focused and illuminated, as will be described herein,despite being taken with the built-in camera found in most mobiletelecommunications devices. The methods (and apparatuses for performingthem) described herein guide a user in taking the images of thepatient's teeth, for example, by displaying an overlay comprising anoutline of teeth in a predetermined view (or sequence of predeterminedviews), on a screen of a mobile telecommunications device, wherein theoverlay is displayed atop the view of the patient's teeth (or other bodyparts, such as face, head, etc.). The user may then move the mobiletelecommunications device so that the overlay approximately matches thepatient's teeth in the view of the patient's teeth and capture (e.g.,manually or automatically by means of the apparatus) an image of theview of the patient's teeth (and in some instances, face and head).

Mobile telecommunications devices can be used to capture dental imagesinstead of using expensive and bulky SLR cameras. FIG. 1A shows the useof a traditional SLR camera 101 to image a patient's teeth 103. Thecamera is typically held by the user (e.g., dentist, orthodontist,dental technician, etc.) and images are taken of the patient's mouthfrom various positions. The user must manually adjust the patientposition and/or camera position. FIG. 1B shows a typical image of ananterior view 105 of a patient's teeth taken with an SLR camera. Inaddition, one or more retractors 107 or other elements may be used toassist in taking the photograph, as shown in FIG. 1C. Any of the methodsand apparatuses described herein may include the use of one or more suchretractors or other assistance devices (including mirrors, probes,etc.), and the apparatuses described herein may prompt the user to placeand/or position such devices, including providing guidance for properpositions.

It may be particularly helpful to adapt a traditional handheld consumerelectronics device, such as phone (e.g., smartphone, smartwatch, pad,tablet, etc.) to take one or, more preferably, a series of images of theteeth at sufficient clarity, e.g., focus, magnification, resolution,lighting, etc. so that these images may be used to track a patient'sprogress and/or pre-screen the patient for a dental or orthodonticprocedure. Thus, such images (and image series) when taken from theproper orientations and with sufficient clarity (focus, magnification,resolution, lighting, etc.) may be used in one or more of: planning adental/orthodontist procedure, determining the feasibility of adental/orthodontic procedure for a particular patient, tracking patientprogress during a dental/orthodontic procedure, determining theeffectiveness of a dental/orthodontic procedure, etc. For example, andof the methods and apparatuses described herein may be used tospecifically track a portion of an orthodontic procedure, including oneor more phases of treatment (such as palatal expansion).

In general, these method and apparatuses may improve on existingtechnology by guiding a user through the process of collecting relevantpatient images (e.g., a predetermined set of images), enhancing and/orconfirming the image quality, associating patient information andtransmitting the set of images so that they may be used in adental/orthodontic procedure.

For example, the methods and apparatuses described herein may use auser's own handheld electronics apparatus having a camera (e.g.,smartphone) and adapt it so that the user's device guides the user intaking high-quality images (e.g., at the correct aspect ratio/sizing,magnification, lighting, focus, etc.) of a predetermined sequence oforientations. In particular, these apparatuses and methods may includethe use of an ‘overlay’ on a real-time image of the screen, providingimmediate feedback on each of the desired orientations, which may alsobe used to adjust the lighting and/or focus, as described herein.

An overlay may include an outline (e.g., a perspective view outline) ofa set of teeth that may be used as a guide to assist in placement of thecamera to capture the patient image. The overlay may be based on ageneric image of teeth, or it may be customized to the user's teeth, orto a patient-specific category (by patient age and/or gender, and/ordiagnosis, etc.). The overlay may be shown as partially transparent, orit may be solid, and/or shown in outline.

FIG. 2A illustrates an example of an overlay 201 comprising an outlineof teeth in a first predetermined view, shown here as an anterior viewof teeth, on a screen 203 of a mobile telecommunications device. FIG. 2Billustrates another example of the overlay 201 in another anterior viewof teeth, on the screen 203 of the mobile telecommunications devicedisplaying a real-time camera image (viewing a dental model of apatient's teeth). As shown in FIGS. 2A and 2B, the on-screen overlay isdisplayed atop the view of the model of the patient's teeth. Theon-screen overlay can help guide a user to move the mobiletelecommunications device to a desired position such that the overlaymatches (or approximately matches, e.g., within 5%, 10%, 15%, 20%, 25%,30%, etc. of the outer or inner edge of the overlay, or a percentage ofthe overlay, e.g., >5%, 10%, 15%, 20%, 25%, 30%, etc.) the view of thepatient's teeth, in this example, shown as a model of the patient'steeth. Therefore, a user can be guided to take dental photos by theon-screen overlay, saving time and effort. As will be described ingreater detail below. The system may be configured to manually,automatically or semi-automatically take the image when the patient'steeth are within the overlay. In FIG. 2A the teeth are shown too farfrom the camera, and do not line up with the overlay; in FIG. 2B theteeth are too close, though they are near an approximate match.

High quality dental images are usually required to submit a dental caseor request a case assessment. The overlay on the screen of the mobiletelecommunications device can increase the quality of dental images byincreasing the accuracy of the alignment of the camera lenses with teethof the patient. The on-screen overlay with the outline of teeth can helpdoctors to take quality dental photos using the mobiletelecommunications device.

The dental photos captured by the mobile telecommunications device canbe uploaded automatically. In general, the methods described herein canincrease the efficiency of the user. Taking dental photos by using themethod disclosed herein can be much faster than using digital cameras.The dental images captured by the method can be of a higher quality andconsistency than simply using a default camera application of a mobiletelecommunications device.

As shown in FIGS. 2A and 2B, users can approximately match visuallyon-screen overlay with the teeth of the patient. The camera of themobile telecommunications device has to be positioned at a rightdistance and a right angle relative to the teeth in order to capture ahigh quality dental image. The user can refer to the on-screen overlayto guide the movement of the mobile telecommunications device camera toset a correct view and distance to the teeth of the patient in the viewof the patient's teeth. The overly can help to precisely align the phonecamera lens with the teeth of the patient and position the mobiletelecommunications device at a right distance and a right angle to theteeth of the patient. For example, the mobile telecommunications devicemay have to be placed close enough to the teeth without being too close.Matching the overlay with the teeth of the patient can greatlyfacilitate the process of alignment. Once the overlay is matching (andthe image focused, e.g., using autofocusing, and particularlyautofocusing within all or a portion of the overly, as described below),the image of the view of the patient's teeth can be captured.

The overlay with an outline of teeth in a predetermined view can furtherprovide information such as required visible teeth in the predeterminedview. For example, FIG. 2C, illustrates a buccal view overlay 209 whichshows what teeth should be visible in the view. In this example, theoverlay may be based on actual patient teeth, or may be generated from atypical patient or synthesized from a population of patients to form anaverage or typical overlay arrangement. In general, an approximatevisual match is needed to select the correct distance, correct angle andcorrect view of the patient's teeth. Also described herein are overlaysthat are based on the patient's approximate dentition. In somevariations the overlay may be progressively refined as the patient'steeth is imaged, so that an increasingly accurate match between theoverlay and the patient's teeth may be made. For example, as an image isfocused, and/or as the camera is brought closer to the patient's teeth,the overlay may be refined to better match the patient's teeth. In somecases a generic patient overlay may not match the patient's teeth evenwhen aligned (e.g., if the patient has gaps or spacing in the teeth,etc.). Thus, the method or apparatus may start with a generic overlayand may thereafter refine the overlay in a manner that is specific tothe patent's teeth. For example, as mentioned herein, the system may betrained to recognize or categorize the teeth and select an appropriateoverlay, or accept an existing overlay as sufficient.

An overlay can be an average (generic) overlay obtained by an artificialmodel. The average overlay may not require specific information from apatient. For example, the average overlay can be obtained from anartificial model which approximately fits most of patients. Users can beguided to approximately match the average overlay with the teeth of apatient. In some other embodiments, the average overlay can comprise aplurality of overlays with a plurality of sizes and types based aplurality of artificial models, for example, overlays for different agegroups, overlays for female and male patients, overlays for patient'shaving an overbite, under bite, etc. In some variations, the users canmanually select an overlay or family of overlays that may fit the teethof the patient and may then be guided to match the selected overlay withthe teeth of the patient. In some variations the method or apparatus mayautomatically select an overlay or family of overlays for the patient.For example, the system may be trained to recognize (e.g., using machinelearning) images corresponding to a particular type of overlay or familyof overlays. Selection may be based on patient information provided bythe user (e.g., patient age, gender, etc.) or based on prior dentalrecord information specific to the patient.

In some other embodiments, the overlay can be a patient-specific overlayderived from the patient's teeth. An example of a customized overlay isshown in FIG. 3A. In some variations the apparatus (e.g.,software/application software) may image the user's teeth or may receivedata derived from a scan, including a digital scan, of the users teeth,and may generate an overlay, such as the outline overlay shown in FIGS.2A-2C. For example, the patient-specific overlay 301 can be obtainedfrom a specific model of teeth for a specific patient and/or from 3Dscan or 3D impressions of the teeth of the patient. This may provideoptimal matching for positioning the handheld electronics (e.g., smartphone) when taking the image(s).

Aligning may be done manually by the user, manually, semi-automatically,or automatically. For example, any of the methods and apparatusesdescribed herein may be used to indicate (e.g., by visual and/or audibleand/or tactile) image that the teeth are aligned in the frame with theoverlay. For example, FIG. 3A illustrates an example of automaticdetection of alignment in real-time with a patient's teeth and anoverlay. In FIG. 3A the overlay (which may be displayed in a firstcolor, such as red) shows lines that illustrate the outline of teeth ina first desired predetermined view (e.g., frontal), when the patient'steeth are not aligned with the overlay. Once the apparatus (e.g.,software/application software executing on the apparatus) determinesthat the teeth are in approximately aligned with (e.g., approximatelymatch) the overlay, as shown in FIG. 3B, the screen may display anindicator, such as, in FIG. 3B, by changing the color of the overlayfrom the first color to a second color (e.g., green, blue, etc.) and/orautomatically taking the image. Note that the method or apparatus mayautomatically determine alignment (a match or approximate match) betweenthe overlay and the patient's teeth in a variety of ways. For example,the method or apparatus may determine alignment when a thresholdamount/percentage of the edges of the patient's teeth in the image arewithin a predetermined minimum distance of the overlay. For example,alignment may be indicated when greater than 50%, greater than 60%,greater than 70%, greater than 80%, greater than 90%, etc. of the edges(e.g., the outer perimeter of the patient's teeth) of the teeth arewithin about 1 mm, about 0.5 mm, about 0.1 mm, about 0.05 mm, etc.). Theedges of the teeth may be determined by any appropriate technique (e.g.,edge detection image processing, etc.).

As mentioned, in general, the overlay may be generic, categorical (e.g.,appropriate for all patients having one or more citatory-relatedcharacteristics) or specific to the patient. The overlay can be atreatment-specific overlay. The teeth of a patient can change. Forexample, the teeth of the patient can change over time including withtreatment. The overlay can be modified according to the predicted oractual changes of teeth of the patient with treatment (e.g., sequentialalignment treatment), thus matching the teeth of the patient moreprecisely. The patient specific overlay or treatment specific overlaycan give user's real-time insight into the treatment progress.

During the imaging procedure, users can view the patient's teeth, forexample, on the screen of the mobile telecommunications device. Userscan further display, on the screen, the overlay comprising the outlineof teeth in a predetermined view, wherein the overlay is displayed atopthe view of the patient's teeth. The overlay can be an average overlay,or a patient specific overlay, or a treatment specific overlay. Userscan move the mobile telecommunications device relative to the patient'steeth so that the overlay approximately matches the patient's teeth inthe view of the patient's teeth. Users can then capture an image of theview of the patient's teeth.

As mentioned, the method can further comprise estimating the quality ofcontour matching between the outline of the overlay and the teeth of thepatient in order to help take high quality photos. When the patient'steeth are located on expected place and at the right angle, the overlayapproximately matches the patient's teeth in the view of the patient'steeth. The method and apparatuses described herein can help preventaccidently confusing view or angle, and may comprise real-time reactionon the image on the screen of the mobile telecommunications device.Interactive contour matching estimation can enable capturing highquality dental images. The method can comprise real-time interactivelyestimating contour matching between the outline of the overlay and theteeth of the patient, for example, by using an indicator. In someembodiments, the methods and apparatuses can automatically detectmatching between the patient's teeth on the screen and the overlay toconfirm the position of the teeth.

Thus, the methods and apparatuses disclosed herein may trigger anindicator when the overlay approximately matches with the patient'steeth. The method or apparatus can automatically capture (or enhancemanual capture of) an image of the view of the patient's teeth when theindicator is triggered as shown in FIG. 3B. Users can move the mobiletelecommunications device relative to the patient's teeth to match theon-screen overlay with the teeth of patient. If the overlay nearlymatches the teeth, the indicator can be triggered, which indicates themoment to capture the image, for example, for progress tracking.Alternatively, in some other embodiments, an image of the teeth can becaptured automatically by the mobile telecommunications device when theindicator is triggered. In some variations, either manual or automatictriggering of the camera to take the image may result in taking aplurality of images in rapid succession. The focus, lighting, and/orexposure time may be adjusted between these images, and the best imagemay be used, or the images may be averaged or combed.

The apparatus (e.g., system) described herein may also provide guidanceby indicating when the image is too far or too close from the camera.For example the camera output may be analyzed to determine theapproximate distance to/from the patient's mouth and this distancecompared to an expected distance to take an optimal image. For example,in any of these variations, image processing may be used to identify thepatient's face (e.g., facial recognition), mouth (e.g., usingmachine-learned feature recognition) or other features, such as thenose, eyes, etc. to determine the distance from the camera and/or whatregions to focus on. Any of the methods and apparatuses described hereinmay include the use (either with an overlay or without an overlay) oftooth recognition. For example, an apparatus may be configured toautomatically detect one or more teeth and trigger an alert to take animage (or automatically take the image) when the identified tooth orteeth are in the desired orientation, size and/or location on thescreen.

As mentioned, the indicator can be triggered by estimating an indicatorof the distance between an edge of the patient's teeth in the view ofthe patient's teeth and the outline of teeth in the overlay. For anotherexample, the indicator can be triggered by estimating an indicator ofthe distance between an edge of the patient's teeth at two or moreregions and the outline of teeth and comparing that indicator to athreshold value. For yet another example, the indicator can be triggeredby estimating an indicator of an average deviation of the outline from acontour of the teeth of the patient. The indicator can be triggered by avariety of ways of estimating a match between the outline of the overlayand the teeth of the patient, not being limited to the examplesillustrated herein. In general, in any of the example described herein,images taken may be associated (including marking, labeling, etc.) withan indicator of the predetermined view and/or user identificationinformation and/or date information. For example an image such as shownin 3B may be taken and marked “frontal”.

The indicator can be a visual indicator, such as a color change asillustrated in FIG. 3B. When the users move the mobiletelecommunications device such that the overlay nearly matches theteeth, the outline of the overlay can change color to green. Forexample, it may be not perfectly match, but each tooth on the contourmay coincide with each real tooth on the screen. When the color of theoutline changes from red to green, it is indicating the overlay nearlymatches the teeth and it is the moment to capture the image.

In some variations the apparatus and methods may include providing oneor more visual or audible clues to the user in aligning the images. Forexample, one or more arrows may be shown on the screen to indicate thatthe user should move the mobile telecommunications device in aparticular direction (or by a particular amount) to align the patient'steeth with the overlay.

For another example, the indicator can be a sound (e.g., tone or voice)which is triggered when the overlay matches the teeth of the patient.For example, the phone can generate a beeping sound and indicate theoverlay matches the teeth and it is the moment to capture the image. Themethod can comprise automatically capturing an image of the patient'steeth when the indicator is triggered and/or permit the user to manuallytake one or more image.

Automatically capturing an image of the patient's teeth when the overlayapproximately matches with the patient's teeth may be triggered by theapparatus. For example, the shutter of the camera of the mobiletelecommunications device can be triggered by an internal indicator.

Generally, the methods and apparatuses described herein are configuredto take a series of predetermined specific views of the teeth. Thus, theapparatus may be an application software (“app”) that guides a user intaking a sequence of specific images. The specific images may be a setthat has particular clinical and/or therapeutic significance, such asfrontal/anterior (mouth closed), frontal/anterior (mouth open), leftbuccal (mouth closed/open), right buccal (mouth closed/open), upper jaw,lower jaw, profile, face, etc.

FIGS. 4A-4H illustrate 8 specific overlay images for the required typesof photos according to the orthodontic standard, including an anteriorview in FIG. 4A, an another anterior view in FIG. 4B, an upper jaw viewin FIG. 4C, a lower jaw view in FIG. 4D, a left buccal view in FIG. 4E,an another left buccal view in FIG. 4F, a right buccal view in FIG. 4G,and an another right buccal view in FIG. 4H.

For example, a method to obtain an image of the teeth may includedisplaying, on the screen, an overlay comprising an outline of teeth inone of an anterior view, a buccal view, an upper jaw view, or a lowerjaw view, wherein the overlay is displayed atop the view of thepatient's teeth. The method and apparatus may walk the user throughtaking a complete or partial set of these images. For example, FIG. 5Ashows an example of a user interface to enable a user to select one of aplurality of overlays for a plurality of dental images in a plurality ofpredetermined views, or to show the user what images it will walk themthrough taking. The predetermined views can include a set of dentalimages according to known orthodontic standards.

In FIG. 5A, eight views (including three external/head or facial views)are included as part of the predetermined set, including a profile view501 of the patient's head, a front view 503 of the patient's head, afront view of the patient's head with the patient smiling 505, a view ofthe upper jaw (from the occlusal surface) 507, a view of the lower jaw(from the occlusal surface) 509, a right buccal view 511 with the jawclosed, an anterior view 513 with the jaw closed, a left buccal view 515with the jaw closed, a right buccal view with the jaw open, an anteriorview with jaw open, and a left buccal view with the jaw open. Theapparatus may walk the user through taking each of these views, and mayallow the user to select which image to take (e.g., in any order) or mayrequire a predetermined order be used. In addition, as shown in FIG. 5B,the apparatus may provide positioning or other guidance to the user intaking the images. FIG. 5B is an example of on-screen message to guide auser to take dental images with a retractor. The image may be a pop-upmessage/reminder, which may be optionally disabled, e.g., forexperienced users. In FIG. 5B, the message 525 indicates that a cheekretractor should be used (e.g., for the intraoral views such as theupper, lower, anterior, buccal views). In some variations, as describedbelow, the apparatus may automatically detect the cheek retractor, andmay use this information to orient the camera, and/or identify thepatient's teeth.

FIG. 5C is an example of a user interface of the apparatus assisting auser in taking a profile image of a patient. In FIG. 5C, the userinterface (on the screen of the mobile telecommunications device) showsan overlay of a profile of a patient's head 531. In FIG. 5D the userinterface shows an overlay 533 of a facial image of a generic patient.The overlay may include one features, such as ears nose, mouth, etc. Inany of these methods and apparatuses facial recognition tools (softwaretools) may be used to help identify and/or confirm the position of apatient's head and/or teeth.

The method can include display on a user interface a plurality ofoverlays for each of a plurality of dental images in a plurality ofpredetermined views. As mentioned above, the user can select one of theoverlays. For example, there can be three different facial images and 8different dental images as shown in FIG. 5A. For each required views ofthe dental images, there can be an open position image and a closedposition image.

The method can further allow the users to take more images of theirchoice. For example, the method can have a user interface to allow auser to input an angle and a distance of his or her choice, thus givingthe user freedom to capture custom images. In some embodiments, themethod can further enable the user to take several photos in motion torestore 3D structure of the teeth.

Any of these methods and apparatuses may further include reviewing thecaptured image on the mobile telecommunications device to confirm imagequality and/or automatically accept/reject the image(s). For example, amethod or apparatus may be configured to check the image quality of thecaptured image and displaying on the screen if the image quality isbelow a threshold for image quality. FIG. 6A is an example of a userinterface having a notification window 606 that indicates the capturedimage is out of focus after reviewing the image quality. FIG. 6B is anexample of a poor quality image of the teeth. In some other embodiments,the method can comprise automatically filtering out images of badquality such as blurred, wrong angle of view, etc. For example, when theimage is too dark or some of all of the teeth are blocked, as shown inFIG. 6B, the image can be excluded and the user required to take analternative image.

In FIG. 6B, the apparatus also indicates that a dental mirror 609 shouldbe used to take the upper jaw image. The apparatus may be configured toautomatically detect when the dental mirror is present, for example, bydetecting a reflection or a symmetry within the image field. In FIG. 6B,the mirror 611 includes an excess of reflected light.

The method can comprise evaluating the image quality and warning doctorsif the image might be rejected. For example, if the image is too dark,or blurry, or out of focus, a warn message is shown on the screen towarn the doctors to retake the photo. The quality of the image can beevaluated and reviewed by a variety of methods. For example, the methodcan comprise evaluating the focus status of the image. For anotherexample, the method can comprise analyzing the image using a library.For yet another example, the method can comprise analyzing thebrightness of the image. In some embodiments, the method can furthercomprise using image recognition algorithms to ensure that all requiredteeth are visible.

FIG. 7A shows an example overlay comprising a cropping frame and anoutline of teeth in an upper jaw view. FIG. 7B shows another exampleoverlay comprising a cropping frame and an outline of teeth in anteriorview. In some embodiments, the method can further comprise displaying,on the screen, an overlay comprising a cropping frame and an outline ofteeth in one of an anterior view, a buccal view an upper jaw view, or alower jaw view, wherein the overlay is displayed atop the view of thepatient's teeth. The method can further comprise automatically croppingthe captured image as indicated by the cropping frame.

Often doctors have to wait one or more days to crop and edit the dentalimages before uploading the images, the method described herein canfurther comprise displaying the overlay with the cropping frame and theoutline of teeth in the predetermined view, and automatically croppingthe captured image. FIG. 7A illustrate a cropping frame 702 in the upperjaw view of teeth. FIG. 7B illustrate another cropping frame 704 in theanterior view of teeth. By matching the overlay, the method canautomatically crop the image to get the correct view of the teeth. Oncethe photo is taken, the doctors can still re-crop and edit later.

FIG. 8A is an example of a block diagram of a method to obtain an imageof teeth of a patient. A user (e.g., dental professional, such as adoctor, orthodontist, etc.) may get quick feedback of treatment progressand/or assessment of treatment suitability. As shown in FIG. 8A, themethod can include (optionally) using patient ID for authorization instep 801. For example, the method can comprise imaging a patient'sidentification using a mobile telecommunications device. The method canfurther comprise automatically populating a form with useridentification information based on the imaged identification.

In general, a method for obtaining a series of images of a patient'steeth can include viewing, on a screen of the mobile telecommunicationsdevice (in real time), the patient's teeth as shown in step 803. Themethod can also include displaying, on the screen, an overlay comprisinga predetermined view and (optionally a cropping frame). The overlay ofthe predetermined view may be, for example: an anterior view (jaws openor closed), a buccal view (left, right, and jaws open/closed) an upperjaw view, or a lower jaw view, etc. The overlay may be displayed atopthe view of the patient's teeth as shown in step 805. As describedherein the method or apparatus performing the method may be configuredto include a selection step for determining which overlay to use (e.g.,by automatic identification of teeth/face shown in the screen and/or byuser manually selecting from a menu, etc.). For example, the method canfurther comprise enabling the user to choose the predetermined view. Auser interface may show the user a predetermined view and a pluralityoverlays for a plurality of predetermined photo views (for example,anterior (open/closed), left buccal (open/closed), right buccal(open/closed), occlusal maxillary (upper jaw), occlusal mandibular(lower jaw), etc.). The user can select one of the plurality overlays tocapture the corresponding dental images. For each predetermined view,the overlay with an outline is shown on the screen of the mobiletelecommunications device. The method can further comprise moving themobile telecommunications device so that the overlay approximatelymatches the patient's teeth in the view of the patient's teeth as shownin step 806. For example, the method can further comprise displayinginstructions about positioning the patient's teeth on the screen of themobile telecommunications device prior to displaying the overlay.Optionally, as described herein, the method may include using theoverlay region to adjust the focus 811, lighting, exposure 813, etc.

In some embodiments, the method can further comprise triggering anindicator when the overlay approximately matches with the patient'steeth as in (optional) step 807. For example, if the overlay is notmatched, it will be displayed using red color. If the overlay matchesthe teeth, the color of the outline changes to green. The method canfurther comprise capturing an image of the view of the patient's teethas shown in step 808. Steps 803 to step 808 can be performed for eachpredetermined photo view 815. The user may take several photos for eachview for more accurate treatment progress estimation. The method canfurther comprise repeating the steps of viewing, displaying, moving andcapturing to capture anterior, buccal, upper jaw and lower jaw images ofthe patient's teeth. In addition, the apparatus may check the image tobe sure that the quality is sufficiently high (e.g., in focus, etc.); ifnot, it may repeat the step for that view.

The method can further comprise transmitting the captured image to aremote server as in step 810 and/or evaluating the captured image formedical treatment by using the set of images collected 809. The captureddental images can be transferred to server part for performing moreprecise estimation of treatment progress and/or for pre-screening apatient. For example, the captured dental images can be used for caseevaluation before starting aligner treatment to evaluate if the patientis a candidate for the treatment as described in FIG. 8B. The captureddental images can also be used to set up case property when doctorsstart the treatment, for example, the captured images can be used tofills prescription forms. The captured dental images can further enabledoctors to track treatment process and give them real time clue orfeedback of treatment process.

In FIG. 8B, a method of remotely pre-screening a patient for anorthodontic treatment may incorporate any of the steps of FIG. 8A. InFIG. 8B, the method first includes guiding a user, with a mobiletelecommunications device having a camera, to take a series of images ofthe patient's teeth in a plurality of predetermined views bysequentially 851, for each predetermined view. As mentioned, an of thesteps described above for FIG. 8A may be used, including but not limitedto: displaying, on the screen, an overlay comprising an outline of teethin one of the predetermined views from the plurality of predeterminedviews, wherein the overlay is displayed atop an image of the patient'steeth 855. Any of these methods may also include capturing the image ofthe patient's teeth when the overlay approximately matches the patient'steeth in the view of the patient's teeth 857. Once the full set ofimages (and any additional information, including cameral position foreach image, label of image, date of series, etc.) are collected, theymay then aggregated into a single set. This information may then betransmitted (e.g., including the series of images) to a remote locationto determine if the patient is a candidate for the orthodontic treatmentbased on the series of images 859. Thereafter, the method may display,on the screen of the mobile telecommunications device, an indicator thatthe patient is, or is not, a candidate for the orthodontic treatment861.

FIG. 9 is an example of a set of facial and dental images of a patientcaptured by using a digital SLR camera. In FIG. 9, the images include 8images: profile, frontal, frontal with smile, upper jaw, lower jaw,right buccal (closed jaws), anterior (closed jaws), and left buccal(closed jaws). FIG. 10 is an example of a set of facial and dentalimages from these same views, captured by using a mobiletelecommunications device through the method disclosed herein. FIGS.11A-11H are examples of a set of dental images captured by using themobile telecommunications device through the method disclosed herein,showing anterior (FIG. 11A), anterior open (FIG. 11B), upper (FIG. 11C),lower (FIG. 11D), right buccal closed (FIG. 11E), right buccal open(FIG. 11F), left buccal closed (FIG. 11G) and left buccal open (FIG.11H). As shown in FIGS. 9-11H, the method disclosed herein can enablethe users to capture high quality dental images in a plurality ofpredetermined views to meet the orthodontic standards.

Targeted Focus

Any of the method and apparatuses described herein may includetooth-specific focusing. In general, the camera of the mobiletelecommunications device may be configured so that the cameraautomatically focuses on a tooth or teeth. In variations in which thesystem is configured to detect a patients teeth within an image, theapparatus may then focus on the teeth automatically. Alternatively oradditionally, the apparatus or method may be configured to use theoverlay, or a region within the overlay, to focus on the patient'steeth. For example, the apparatus (e.g., a non-transitory,computer-readable storage medium storing a set of instructions capableof being executed by a processor of a mobile telecommunications devicehaving a camera) may be configured to focus on a subset of the regionwithin the overlay on the display, and automatically focus within thisregion or sub-region. Within this region or sub-region, the camera maybe controlled to perform any appropriate type of autofocusing, includingbut not limited to: contrast-detection auto-focus, phase-detectionauto-focus, and laser auto-focus.

For example, the apparatus or method may first disable the camera'snative autofocusing, which may default on a particular region (e.g., thecentral region), motion detection, and/or object (e.g., face)recognition, or some variation of these. The native autofocusing, ifused with the overlay and method of matching the overlay to a patient'steeth may instead focus on the lips, cheek retractors, tongue, etc.,rather than the teeth or a portion of the teeth. By instead restrictingthe autofocusing to a region that is limited to the overlay or a portionof the overly, the apparatus and method may instead properly focus onthe teeth.

For example, FIG. 12A shows an example of a screen 1201 having anoverlay 1203 shown atop any images from the camera being displayed onthe screen. In FIG. 12A, a region (point) of focus within the overlay isshown as a dot 1205 within the overlay of one of the upper incisors.Thus, in this example, the point of focus on the screen is shown as thefront (incisor) teeth when the teeth are approximately aligned with theoverlay. FIG. 12 shows an example of an overlay for an anterior (openmouth) view. Other views may have a point of focus within the overlaycentered on different teeth, or more than one tooth. For example, ananterior view may a have a point of focus within the overlay over abicuspid tooth, or a visible molar, and/or a canine. The user (e.g.,doctor, orthodontist, technician, assistant, etc.) may align thepatient's teeth with the overlay, and take an image and the mobiletelecommunications device may be controlled so that a region within theoverlay is automatically controlled to focus within this region of theoverlay. In FIG. 12A, and exemplary image shown in FIG. 12C, the focuswas limited to on one of the central incisor teeth within the overlay.FIGS. 12B and 12C provide a comparison between an image taken with thepatient's teeth generally aligned within an overlay for an anterior(bite closed) image 1211 with the camera for the mobiletelecommunications device set to its default autofocusing technique(shown in FIG. 12B, on the left) compared to the targeted autofocusingdescribed above, in which the autofocusing was limited by the apparatusto a region 1213 corresponding to an incisor within the overlay (shownin FIG. 12C, on the right).

Adapted Lighting Mode

Any of the apparatuses and methods described herein may also beconfigured to automatically and accurately adjust the lighting and/orexposure time so that the teeth are optimally illuminated for imaging,so that details may be apparent. As described above for the autofocusingwithin the image, the illumination (lighting) may be similarly adjustedby using the overlay or a sub-region of the overlay to set the intensityof the applied lighting, such as the flash.

In general, a camera of a mobile communications device may include alight source providing illumination when taking an image. The camera mayhave one or more lighting modes for operation, including, for example, abust flash (a pulse or flash of light correlating with image capture),torch (a continuously on light), or no flash/no illumination (e.g., notproviding illuminated). Any of the methods and apparatuses describedherein may improve the images of the patient's teeth that are capturedby automatically selecting and controlling a particular lighting modefor each image captured (e.g., each predetermined view). For example, inparticular, the apparatus may be configured by adjusting or controllingthe lighting mode so that no flash is used when taking the facial images(e.g., profile facial images, frontal facial images, with or withoutsmile, etc.); the lighting mode may be set to burst flash when takingthe occlusal images (e.g., upper occlusal/upper jaw and lowerocclusal/lower jaw); and torch illumination may be automaticallyselected when taking intra oral photos (e.g., anterior views, buccalviews etc.). The intensity of the flash may also be adjusted. Forexample, the intensity of the light applied may be adjusted based on alight level detected from the region of an image within an overlay onthe screen. In some variations the choice to use any additionalillumination at all may be made first, based on the light intensitywithin the overlay region; if the light level is below a threshold(e.g., within the lower 5%, 10%, 15%, 20%, 25%, 30%, etc. of the dynamicrange for intensity of light for the camera of the mobiletelecommunications device) within all or a portion of the overly, thenthen lighting mode may be selected based on the type of predeterminedview for the overlay. For example, if the overlay shows an intra oralview (e.g., anterior, anterior mouth open, buccal, buccal mouth open,etc.) then the lighting mode may be set to torch, and in some cases thelevel of brightness of the torch may be adjusted based on the lightlevel detected within the overlay. If the overlay corresponds to anocclusal view (e.g., upper jaw, lower jaw) then then lighting mode maybe set to burst flash, and in some cases the level of brightness of thetorch may be adjusted based on the light level detected within theoverlay.

FIGS. 13A-13B and 13C-13D illustrate comparisons between images of theteeth taken guided by an overlay (not visible on the images shown). FIG.13A shows an upper jaw, occlusal view, taken with a dental mirror usingthe default focusing and with no flash. For comparison, FIG. 13B showsthe same teeth with the focus limited to a sub-region of the overlay(not shown) and the flash on as a bursting flash. In FIG. 13B,additional details of the teeth 1307 in the upper jaw, as seen with thedental mirror 1305, as visible. FIGS. 13C and 13D show a similarexample, in which the lower jaw 1309 is imaged without limiting thefocus to the overlay region (FIG. 13C) and without a flash (e.g., flashin the ‘off’ mode). For comparison the same predetermined view of theteeth is shown in FIG. 13D with the focus adjusted to a region withinthe overlay for the lower teeth and with the flash automaticallyadjusted to be on (torch mode). A dental mirror was used to capture thisimage as well.

Thus, any of the methods and apparatuses described herein mayautomatically adjust the illumination provided by switching on or offthe automatic flash and/or by setting the light level, or allowing theuser to adjust the light level(s). For example, the method of apparatusmay toggle between automatic flash, no flash/illumination, user-adjustedlight level (“torch mode”) or automatically adjusted light level, basedon one or more of user preferences and the image to be taken. Forexample, when taking a profile or face image, the apparatus may beconfigured so that the flash is turned on, e.g., defaulting to thecamera's auto-flash function, if one is present. When taking anintraoral images (e.g., an anterior view, a buccal view an upper jawview, or a lower jaw view, etc.) the flash may be turned off, andinstead the apparatus may be adjusted to use a “torch” mode, in whichthe light is continuously on, particularly when imaging. The level ofthe light may be set automatically, as mentioned above, or it may beadjusted by the user. For example, when taking intraoral images of theteeth, the torch function of the camera light may be set to be on at alevel that is approximately 10-50% of the peak intensity (e.g., between15-40%, between 20-30%, etc.). In some variations the torch intensitymay be adjusted in real time based on the image quality, and inparticular, based on the light level of a region within the overlay(e.g., centered on or near the molars). Alternatively or additionally,the user may manually adjust the light intensity in the torch mode,e.g., by adjusting a slider on the screen of the device or by one ormore buttons on the side of the device.

Similarly, the exposure time for the camera may be adjusted based on theamount of light within all or a portion of the overly region of theimaging field. Controlling the image exposure and/or the depth of fieldscope for the image may be guided by the region within the overlay. Forexample, the apparatus (e.g., a non-transitory, computer-readablestorage medium storing a set of instructions capable of being executedby a processor of a mobile telecommunications device having a camera)may control the camera so that any auto-exposure function of the camerais modified to base the exposure on a targeted point or region withinthe overlay. The depth of field may also be adjusted by the apparatus.

The exposure time may be set automatically based on a region within theoverlay, as mentioned above. The region within the overlay used forsetting the exposure may be different from a region used to set eitherthe focus or the light intensity. In addition, these regions may bedifferent for different views. For example, in anterior images, thefocus may be set using a region within the overlay that is on the front(e.g. incisor) teeth, as shown in FIG. 12A, while the region for settingthe illumination (e.g., light intensity) may be set based on moreanterior teeth (e.g., molars, premolars, etc.) and the region used forsetting the exposure may be the same as either the region for settingthe focus, the same as the region for setting the illumination, or maybe different from either of these. In some variations, all three regionsmay be the same.

Cheek Retractor Detection

Any of the methods and apparatuses (e.g., systems) described herein mayinclude a check retractor and/or the automatic detection of a cheekretractor. For example, the apparatuses and methods described herein mayavoid having a user take a patient's images of the predetermined viewswithout using a cheek retractor, particularly when the predeterminedviews benefit from the use of a cheek retractor. FIG. 14A illustrates anexample of a right buccal view of a patient manually retracting (using afinger) their right cheek. Such images may be useful, and indeedmultiple such images may be taken and combined (by image stitching)together, e.g., by pulling or manually retracting the cheek fromdifferent angles and combining the different images to show as much ofthe buccal view as possible, including the upper and lower teeth.However, in some variations, it may be beneficial to use a mechanicalcheek retractor, such as the one shown in FIG. 14C. This may allow moreof the teeth to be visible than would be possible either manuallyretracting the checks (as shown in FIG. 14A) or simply pulling the lipsand cheeks back using the facial muscles, as shown in FIG. 14B.

The apparatuses and methods described herein may remind the user to usecheek retractors before taking intra-oral photos, as described above(e.g., using an alert window, etc.). However, the user (who may be thepatient) may choose not to, leading to images for the intra-oralpredetermined views in particular that are not optimal, and may beharder to process. In some variations, the apparatus or method mayautomatically detect if a check retractor is present and may warn theuser when they are not detected. For example machine learning may beused to train the apparatus to automatically detect a cheek retractor.Alternatively or additionally, the cheek retractor may include one ormore markings on the check retractor in visible regions that may bereadily detected. The markings (e.g., dots, bar codes, QR codes, text,patterns, icons, etc.) may be detected by the apparatus and may also beused to help automatically identify the position of the patient, andtherefore which predetermined view (e.g., which predetermined viewoverlay) should be used or is being taken.

In the exemplary device shown in FIG. 14C, the cheek retractor 1408includes a plurality (in this example, six) of markings 1406, 1406′,1406″, 1406′ on the outer surface that will be visible when imaging thepatient wearing the cheek retractor. The markings may be on the outerand upper surface, which will be over the patient's lips when worn. Theunitary cheek retractor shown has a left side with a patient handle 1409that is connected via a flexible bridge region 1413 to a right side witha second patient handle 1411. The patient-facing side includes a channelor groove for holding the lips and cheeks and the bridge region mayapply a force to separate the left and right side, holding the cheeksopen, exposing the teeth. The cheek retractor may be colored (e.g., mayinclude color dots) for identifying the orientation of the patient'steeth when worn. In the example shown in FIG. 14C, two of the markings1406, 1406′ have different orientation markers, shown as a horizontalline on one 1406 and a vertical line on the other 1406′ (the line may beformed by the flat edge of the semicircular shape forming the marking).These orientation markers may be formed in to the device, or they may beprinted on it. I some variations the markings may be different colorsand/or shapes, and may therefore provide orientation and/or distanceinformation. For example, the marking(s) may be used to determine thedistance between the patient (or the retractor device) and the camera.As mentioned herein, the method and/or apparatus may determine anestimate of distance from the size of an individual marker and/or fromthe spacing between different markings in the resulting image. Ingeneral, it may be desired to take any of the intraoral images with thecamera between about 6 and about 16 inches from the patient's mouth(e.g., between about 7 and 14 inches, between about 8 and 12 inches,etc.). As mentioned, if the distance is outside of this predefinedrange, the apparatus may instruct the user to adjust the position (e.g.,move closer or farther, etc.). Similarly, the apparatus may center theimage using the markings, so that the teeth (or a portion of the teeth)are centered within the image(s).

The markings on the retractor may also be used to aid in automaticallycropping the images.

Any of the apparatuses and methods described herein may also estimatebetween the patient and the camera of the mobile telecommunicationsdevice. For example, any of these methods and apparatuses may facialdetection from the image to identify the patient's face; onceidentified, the size and position of the face (and/or any landmarks fromthe patient's face, such as eyes, nose, ears, lips, etc.) and maydetermine the approximate distance to the camera. This information maybe used to guide the user in positioning the camera; e.g., instructingthe user to get closer or further from the patient in order to take theimages as described above. For example, in FIG. 15, the box 1505 on thescreen identifies the automatically detected face/head of the patient;the face was identified using the face identification software availablefrom the developer. FIG. 16 shows another example of faceidentification. In both examples, the method and apparatus may determinethat the camera is too far away by comparing the size of the recognizedface (the box region) to the size of the field. In both cases, the usermay be provided instructions (audible, visual, text, etc.) to move thecamera closer. In FIG. 15, the image to be taken is an anterior view ofthe teeth, and the user may be instructed to move much closer to focuson the teeth. In FIG. 16, the view may be a face (nonsmiling) image ofthe patient, and the user may be instructed to get closer. The distanceto the patient may be estimated in any appropriate manner. For example,the distance may be derived from the size of the rectangle made aroundthe patient's face or mouth, as mentioned (e.g., if it is too small, thecamera is too far). The size of the ‘box’ identifying the face may beabsolute (e.g., must be above a set value) or as a percent of the imagefield size.

The distance from the camera may be approximate using other identifiedfeatures, as mentioned, including the eye separation distance (pupilseparation distance, etc.) when these features are visible. Any of thesedistance estimates may be stored with the image for later use, e.g., inestimating size or projecting three-dimensional features.

Also described herein is the use of continuous imaging (shooting) of theteeth. For example, rather than taking individual images, e.g., one at atime, the apparatus or method may be configured to general patientphotos by using a continuous shooting mode. A rapid series of images maybe taken while moving the mobile device. Movement can be guided by theapparatus, and may be from left to right, upper to lower, etc. From theuser's perspective it may be similar to taking a video, but a series ofimages (still images) may be extracted by the apparatus. For example,the apparatus may automatically review the images and match (orapproximately match) views to the predetermined views, e.g., using theoverlays as described above. The apparatus may select only those imageshaving a sufficiently high quality. For example, blurry, dark or notoptimally positioned photos may be automatically rejected. Similarly,multiple photos may be combined (by stitching, averaging, etc.).

In some variations, the user may be patient, and the apparatus may beconfigured to allow the user to take continuous “selfies” in thismanner. For example, a continuous shooting mode may allow the patient totake photos of their smile alone with the camera (e.g., a back-facingcamera). The apparatus may use face and/or smile detection to guidepatient and indicate if the camera is well positioned. In variations inwhich the screen is not facing the user, the apparatus may be configuredto provide a user-detectable output (e.g., a flash, sound or the like)indicating that the patient should start moving the device around theirhead/mouth and may indicate that they should move the camera closer orfurther, to the right/left, up/down, etc. For example, an indicator suchas light (e.g., flash) or sound (a voice or tone, etc.) may be used asan indicator. For example, a flashing and/or sound may be used toindicate to patient when to start moving mobile device, and theapparatus may start taking photos in the continuous mode (also referredto as a burst mode) and move the mobile device in the indicateddirection.

As mentioned above, any of these variations may include detection of theteeth automatically, e.g., by machine learning. Detection of the patientteeth automatically may improve photo quality. In some variations, themachine learning (e.g., the machine learning framework provided by Applewith iOS 11) may be used to detect the presence of teeth when photos aretaken, and to further guide the user. For example, the user may bealerted when the teeth are not visible, or to automatically select whichpredetermined view overly to use, to indicate if the angle is notcorrect, to indicate that the user is too close or too far from thepatient, etc.

In general, in addition to the images captured by the apparatus,additional information, including the orientation and location of thecamera relative to the patient, may be extracted from the sequence ofimages taken from a continuous shooting described above. For example,positional information, including the relative distance and/or anglethat the camera is relative to the patient's mouth, may be extractedfrom the time sequence. Additional information, such as motion sensors(e.g., accelerometers, etc.) in the mobile telecommunications device,may be used as well. This additional sensor information (e.g.,accelerometer, angle, etc.) information may be provided with the images.Such information may be helpful in both guiding the user, e.g., ininstructing the user to get closer/farther, move more slowly, etc., butalso in calculating dimensional information (e.g., size,three-dimensional position and/or surface or volumetric information,etc.).

Pre-Screening for Orthodontic Treatment

As mentioned above, in general, the apparatuses and methods describedherein may be used to remotely pre-screen a patient for an orthodontictreatment. For example, the methods described herein may include, and/ormay be used to determine if a patient would benefit from an orthodonticcorrection procedure to orthodontically move and straighten thepatient's teeth using, e.g., a series of dental aligners. These methodsand apparatuses may be part of a case assessment tool. The user (adental professional or in some cases the potential patient) may beguided to take a set of predetermined views as described above, andthese views may be transmitted (e.g., uploaded) from the mobiletelecommunications device to a remote location. At the remote location,which may include a server, the images may be processed manually,automatically or semi-automatically to determine if the patient is acandidate for the orthodontic procedure.

The series of predetermined views described herein may be used todetermine if a patient is a good candidate by identifying (manually orautomatically) the amount and extent of movement of teeth required inorder to straighten the teeth. Patients requiring excessive toothmovement may be indicated as not a candidate. Patient requiring surgicaltreatment (e.g., patients requiring palatal expansion, etc.) may beindicated as not a candidate. In some variations patients requiringtooth extraction and/or interproximal reduction may be indicated as nota candidate, at least for some orthodontic treatments. In variations,rather than simply determining that a patient is a candidate or not acandidate for a particular orthodontic treatment, the apparatus ormethod may instead indicate which type of orthodontic treatment would bebest.

These methods may be used to pre-screen for any type of orthodontictreatment, as mentioned, including (for example), teeth straighteningusing one or a series of temporary aligners (e.g., which may be changedregularly, e.g., weekly). The type of orthodontic treatment may belimited to relatively easy orthodontic straightening procedures, such asorthodontic treatments with aligners that may take less than x months tocomplete (e.g., 1 month or less, 2 months or less, 3 months or less, 4months or less, etc.).

Either before, during or after capturing the series of images, any ofthese methods and apparatuses may be configured to collect informationabout the patient, as discussed above. In addition to, or instead of,patient-identification information, the apparatus may also includeinformation about the patient and/or user's chief orthodontic concernfor the patient's teeth (e.g., tooth crowding, tooth spacing, smilewidth/arch width, smile line, horizontal overjet, vertical overbite,cross bite, bite relationship, etc.). The apparatus may include a menuof these concerns and may allow the user (dental professional and/orpatient) to select one or more of them, or enter their own. The one ormore chief dental concerns may be added to the set or series of imagesfrom the predetermined views. For example, the chief dental concern(s)may be appended or combined with the set or series of images, andtransmitted to the remote site and used to determine if the patient is agood candidate for a particular orthodontic procedure.

In general, the images may be used to quantify and/or model the positionof the patient's teeth. The position and orientation of the patientsteeth, relative to the other teeth in the dental arch or the oppositedental arch, may provide an estimate of the movement or proceduresnecessary to correct (e.g., align) the patients teeth, or in somevariations, the progress of an ongoing treatment.

The methods described herein, may also include monitoring a patientundergoing an orthodontic treatment. For example, the steps of guidingthe user, with the same or a different mobile telecommunications devicehaving a camera, to take a series of images of the patient's teeth fromthe plurality of predetermined views (e.g., by sequentially displaying,on a screen of the mobile telecommunications device, an overlaycomprising an outline of teeth in each of the predetermined views) maybe repeated during an orthodontic treatment in order to monitor thetreatment. Images taken prior to treatment may be compared with imagestaken during or after treatment.

In any of the methods an apparatuses described herein, the images may beuploaded to a remote server, or storage facility, or they may be keptlocal to the mobile telecommunications device. When maintained locallyon the mobile device, any copies sent remotely for analysis may bedestroyed within a predetermined amount of time (e.g., after analysis iscomplete), so that no additional copies are sent. The images and anyaccompanying information may generally be encrypted.

As mentioned, any of the methods and apparatuses described herein may beconfigured for automatic detection of a mirror, such as a dental mirror,used to take any of the images. For example, the apparatus may beconfigured to identify that the image is a reflection, or to identify amarker on the mirror. A reflection may be determined by identifying adiscontinuity (e.g., a line) at the edge(s) of the mirror, and/or themirror-imaged/inverted images of parts of the image, such as the teeth.When a mirror is detected, the apparatus may display a mirror icon (ormay indicate on the mirror icon. In some variations the image resultingmay be inverted (mirrored) so that the image is in the same orientationas it would be had a mirror not been used.

The images (e.g., the predetermined series of images) may be used tosupplement additional information (e.g., scans, 3D models, etc.) of thepatient's teeth. Images taken as described herein may provideinformation on the shape, location and/or orientation of the patient'steeth and gingiva, including information related to the patient's root.Thus, this information may be used in conjunction with other images ormodels, including 3D models (e.g., digital models) of the patient'steeth, and may be combined with, or may supplement this information.

Various embodiments of the disclosure further disclose a non-transitory,computer-readable storage medium storing a set of instructions capableof being executed by a processor of a mobile telecommunications device,that, when executed by the processor, causes the processor to displayreal-time images of the patient's teeth on a screen of the mobiletelecommunications device, display an overlay comprising an outline ofteeth in a predetermined view atop the images of the patient's teeth,and enable capturing of an image of the patient's teeth. For example,the non-transitory, computer-readable storage medium, wherein the set ofinstructions, when executed by the processor, can further cause theprocessor to display a generic overlay. For another example, thenon-transitory, computer-readable storage medium, wherein the set ofinstructions, when executed by the processor, can further cause theprocessor to display a patient-specific overlay derived from thepatient's teeth.

For example, the non-transitory, computer-readable storage medium,wherein the set of instructions, when executed by the processor, canfurther cause the processor to automatically trigger an indicator whenthe overlay approximately matches with the patient's teeth. For example,the non-transitory, computer-readable storage medium, wherein the set ofinstructions, when executed by the processor, can further cause theprocessor to estimate an indicator of the distance between an edge ofthe patient's teeth in the view of the patient's teeth and to triggerthe indicator when the outline of teeth in the overlay is less than orequal to a threshold value. For example, the non-transitory,computer-readable storage medium, wherein the set of instructions, whenexecuted by the processor, can further cause the processor to estimatean indicator of the distance between an edge of the patient's teeth attwo or more regions in the view of the patient's teeth and to triggerthe indicator when the outline of teeth in the overlay is less than orequal to a threshold value. For example, the non-transitory,computer-readable storage medium, wherein the set of instructions, whenexecuted by the processor, can further cause the processor to triggerthe indicator by displaying a visual indicator on the screen.

In general, various embodiments of the disclosure further disclose anon-transitory, computer-readable storage medium storing a set ofinstructions capable of being executed by a processor of a mobiletelecommunications device, that, when executed by the processor, causesthe processor to display real-time images of the patient's teeth on ascreen of the mobile telecommunications device and display an overlaycomprising a cropping frame and an outline of teeth in one of ananterior view, a buccal view an upper jaw view, or a lower jaw view,wherein the overlay is displayed atop the images of the patient's teeth,and enable capturing of an image of the patient's teeth. Thenon-transitory, computer-readable storage medium, wherein the set ofinstructions, when executed by the processor, can further cause theprocessor to review the captured image and indicate on the screen if thecaptured image is out of focus and automatically crop the captured imageas indicated by the cropping frame.

For example, the non-transitory, computer-readable storage medium,wherein the set of instructions, when executed by the processor, canfurther cause the processor to check the image quality of the capturedimage and displaying on the screen if the image quality is below athreshold for image quality. For example, the non-transitory,computer-readable storage medium, wherein the set of instructions, whenexecuted by the processor, can further cause the processor toautomatically crop the captured image based on a cropping outlinedisplayed as part of the overlay. For example, the non-transitory,computer-readable storage medium, wherein the set of instructions, whenexecuted by the processor, can further cause the processor to transmitthe captured image to a remote server.

For example, the non-transitory, computer-readable storage medium,wherein the set of instructions, when executed by the processor, canfurther cause the processor to display an overlay comprising an outlineof teeth in a predetermined view such as an anterior view, a buccal viewan upper jaw view, or a lower jaw view. For example, the non-transitory,computer-readable storage medium, wherein the set of instructions, whenexecuted by the processor, can further cause the processor to repeat thesteps of viewing, displaying, moving and capturing to capture anterior,buccal, upper jaw and lower jaw images of the patient's teeth. Forexample, the non-transitory, computer-readable storage medium, whereinthe set of instructions, when executed by the processor, can furthercause the processor to capture an image of a patient's identificationusing the mobile telecommunications device and automatically populate aform with user identification information based on the imagedidentification. For example, the non-transitory, computer-readablestorage medium, wherein the set of instructions, when executed by theprocessor, can further cause the processor to display instructions onpositioning the patient's teeth on the screen of the mobiletelecommunications device prior to displaying the overlay.

The systems, devices, and methods of the preferred embodiments andvariations thereof can be embodied and/or implemented at least in partas a machine configured to receive a computer-readable medium storingcomputer-readable instructions. The instructions are preferably executedby computer-executable components preferably integrated with the systemincluding the computing device configured with software. Thecomputer-readable medium can be stored on any suitable computer-readablemedia such as RAMs, ROMs, flash memory, EEPROMs, optical devices (e.g.,CD or DVD), hard drives, floppy drives, or any suitable device. Thecomputer-executable component is preferably a general orapplication-specific processor, but any suitable dedicated hardware orhardware/firmware combination can alternatively or additionally executethe instructions.

EXAMPLES

In one example, described herein is a photo uploading mobile app(control software) that may be installed on a mobile device such as asmartphone and may control the smartphone camera to take dental imagesof a patient in a particular manner. The particular manner may include aspecific sequence of photos, as well as controlling the image qualityand/or imaging characteristics in order to more accurately plan or tracka therapeutic treatment, e.g., of a series of dental aligners.

For example, the application (“app”) may be configured to require login(username, password) in order to use. Alternative logins (e.g.,fingerprint, passcode, etc.) may be used to login. Once logged in, theapp may present a list of patients and/or may allow patients to beadded. The user may select or add a patient, e.g., from a selectablelist that is displayed. Adding a patient may include enteringidentifying information about the patient (e.g., first name, last name,date of birth, location such as country, city, state, zip code, etc.,and gender. Thereafter, the app may guide the doctor or clinician intaking a predetermined series of images, such as those shown in FIG. 5A,and described above.

For existing patients, the app may allow the user to view photos thatwere already taken and/or begin taking new photos. Thus, any of theapparatuses (including apps) described herein may allow the user to takethe requested series of images (which may depend on the patientidentity, treatment plan, etc.) and may include taking additional images(e.g., 3 or more, 4 or more, 5 or more, 6 or more, 7 or more, 8 or more,9 or more, 10 or more, 11 or more, 12 or more, etc.). For each patient,photos may be taken at different times, showing the progression oftreatment. For example, a series of photographs (such as those shown inFIG. 5A) may be taken prior to treatment, and one or more times duringtreatment, and/or post-treatment. Additional photos may be taken at anyof these times as well. When the user is taking photographs in theseries, the app may guide the user in taking patient photos from thepredetermined series, and may display on the screen one or more outlines(including outlines of a typical set of teeth, head/face, etc., asdescribed above). The app may also guide the user in applying assistancedevices, such as a retractor and/or dental mirror. Once the image istaken, the app may be pre-cropped and/or cropped manually by the user.The app may also examine the content of the image to confirm focus,subject matter (angle, etc.) illumination, etc. The app may prompt theuser to re-take and/or save the image(s).

Images/photos taken may be uploaded by the apparatus into a remoteserver or site, and stored for later retrieval by the physician and/ortransmission to a third party along with patient and/or physicianidentifying information, and/or information about when they were taken.The physicians may approve or review the images using the app, or usinganother computer accessing the remote site. Images may be individuallyuploaded, or they may be uploaded as a composite of multiple separateimages.

A physician may also comment and/or append comments, on the images or toaccompany the images. In any of these apparatuses, the app may alsoprovide access to patient case management software including modelingand 3D images/rendering of the patient's teeth.

The app may also include instructions (e.g., a frequently askedquestions portion, a tutorial, etc.).

In some variations, the physician may mark or otherwise signify that aparticular patient or case be processed by a remote server, e.g., forclinical assessment, and/or to prepare a series of aligners. Asmentioned above, the app may be used on any mobile device having orcommunicating with a camera, such as a smartphone, smartwatch, pad, etc.

Although the examples described herein are specifically described asbeing for use with a mobile telecommunication device (such as asmartphone, pad, etc.), in some variations these methods and apparatusesimplementing them may be performed with devices that include display anda processor that ae not limited to mobile telecommunications devices.For example, the methods and apparatuses may be configured for use witha virtual reality (VR)/augmented reality (AR) headset or any otherimaging device that may transmit images (e.g., photos) directly to acomputer via a direct connection (e.g., cable, dedicated wirelessconnection, etc.), and/or may save the images to removable ortransferrable memory (e.g., an SD card); the image data could then beuploaded to a remote server via another device. Thus, any of the methodsand apparatuses described herein which recite or refer to a mobiletelecommunications device may be performed with an imaging device.

For example, a method for remotely pre-screening a patient for anorthodontic treatment may include: guiding a user, with an imagingdevice having a camera, to take a series of images of the patient'steeth in a plurality of predetermined views, transmitting the series ofimages from the imaging device to a remote location to determine if thepatient is, or is not, a candidate for the orthodontic treatment basedon the series of images; and displaying, on a screen, an indicator thatthe patient is, or is not, a candidate for the orthodontic treatment.

Similarly, a method for remotely pre-screening a patient for anorthodontic treatment may include: guiding a user, with an imagingdevice having a camera, to take a series of images of the patient'steeth from a plurality of predetermined views by sequentiallydisplaying, on a screen of the imaging device, an overlay comprising anoutline of teeth in each of the predetermined views; receiving, in theimaging device, an indication of the patient's chief dental concern;aggregating, in the imaging device, the series of images and the chiefdental concern; transmitting the aggregated series of images and thechief dental concern to a remote location to determine if the patient isa candidate for the orthodontic treatment based on the series of images;and displaying, on the screen of the imaging device or a device incommunication with the imaging device, an indicator that the patient is,or is not, a candidate for the orthodontic treatment.

As another example, a system for remotely pre-screening a patient for anorthodontic treatment, may include: a non-transitory, computer-readablestorage medium storing a set of instructions capable of being executedby a processor of an imaging device having a camera, that, when executedby the processor, causes the processor to: guide a user to take a seriesof images of the patient's teeth in a plurality of predetermined viewswith the camera; transmit the series of images from the imaging deviceto a remote location to determine if the patient is a candidate for theorthodontic treatment based on the series of images; and display, on ascreen of the imaging device or a device in communication with theimaging device, an indicator that the patient is, or is not, a candidatefor the orthodontic treatment.

When a feature or element is herein referred to as being “on” anotherfeature or element, it can be directly on the other feature or elementor intervening features and/or elements may also be present. Incontrast, when a feature or element is referred to as being “directlyon” another feature or element, there are no intervening features orelements present. It will also be understood that, when a feature orelement is referred to as being “connected”, “attached” or “coupled” toanother feature or element, it can be directly connected, attached orcoupled to the other feature or element or intervening features orelements may be present. In contrast, when a feature or element isreferred to as being “directly connected”, “directly attached” or“directly coupled” to another feature or element, there are nointervening features or elements present. Although described or shownwith respect to one embodiment, the features and elements so describedor shown can apply to other embodiments. It will also be appreciated bythose of skill in the art that references to a structure or feature thatis disposed “adjacent” another feature may have portions that overlap orunderlie the adjacent feature.

Terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention.For example, as used herein, the singular forms “a”, “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“comprises” and/or “comprising,” when used in this specification,specify the presence of stated features, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, steps, operations, elements, components, and/orgroups thereof. As used herein, the term “and/or” includes any and allcombinations of one or more of the associated listed items and may beabbreviated as “/”.

Spatially relative terms, such as “under”, “below”, “lower”, “over”,“upper” and the like, may be used herein for ease of description todescribe one element or feature's relationship to another element(s) orfeature(s) as illustrated in the figures. It will be understood that thespatially relative terms are intended to encompass differentorientations of the device in use or operation in addition to theorientation depicted in the figures. For example, if a device in thefigures is inverted, elements described as “under” or “beneath” otherelements or features would then be oriented “over” the other elements orfeatures. Thus, the exemplary term “under” can encompass both anorientation of over and under. The device may be otherwise oriented(rotated 90 degrees or at other orientations) and the spatially relativedescriptors used herein interpreted accordingly. Similarly, the terms“upwardly”, “downwardly”, “vertical”, “horizontal” and the like are usedherein for the purpose of explanation only unless specifically indicatedotherwise.

Although the terms “first” and “second” may be used herein to describevarious features/elements (including steps), these features/elementsshould not be limited by these terms, unless the context indicatesotherwise. These terms may be used to distinguish one feature/elementfrom another feature/element. Thus, a first feature/element discussedbelow could be termed a second feature/element, and similarly, a secondfeature/element discussed below could be termed a first feature/elementwithout departing from the teachings of the present invention.

As used herein in the specification and claims, including as used in theexamples and unless otherwise expressly specified, all numbers may beread as if prefaced by the word “about” or “approximately,” even if theterm does not expressly appear. The phrase “about” or “approximately”may be used when describing magnitude and/or position to indicate thatthe value and/or position described is within a reasonable expectedrange of values and/or positions. For example, a numeric value may havea value that is +/−0.1% of the stated value (or range of values), +/−1%of the stated value (or range of values), +/−2% of the stated value (orrange of values), +/−5% of the stated value (or range of values), +/−10%of the stated value (or range of values), etc. Any numerical rangerecited herein is intended to include all sub-ranges subsumed therein.

Although various illustrative embodiments are described above, any of anumber of changes may be made to various embodiments without departingfrom the scope of the invention as described by the claims. For example,the order in which various described method steps are performed mayoften be changed in alternative embodiments, and in other alternativeembodiments one or more method steps may be skipped altogether. Optionalfeatures of various device and system embodiments may be included insome embodiments and not in others. Therefore, the foregoing descriptionis provided primarily for exemplary purposes and should not beinterpreted to limit the scope of the invention as it is set forth inthe claims.

The examples and illustrations included herein show, by way ofillustration and not of limitation, specific embodiments in which thesubject matter may be practiced. As mentioned, other embodiments may beutilized and derived there from, such that structural and logicalsubstitutions and changes may be made without departing from the scopeof this disclosure. Such embodiments of the inventive subject matter maybe referred to herein individually or collectively by the term“invention” merely for convenience and without intending to voluntarilylimit the scope of this application to any single invention or inventiveconcept, if more than one is, in fact, disclosed. Thus, althoughspecific embodiments have been illustrated and described herein, anyarrangement calculated to achieve the same purpose may be substitutedfor the specific embodiments shown. This disclosure is intended to coverany and all adaptations or variations of various embodiments.Combinations of the above embodiments, and other embodiments notspecifically described herein, will be apparent to those of skill in theart upon reviewing the above description.

What is claimed is:
 1. A non-transitory, computer-readable storage mediahaving executable instructions that are executed by a processor of amobile telecommunications device to cause the mobile telecommunicationsdevice having a camera to perform a method for capturing dental imagesfor analysis, the method comprising: guiding a patient to capture, withthe camera of the mobile telecommunications device, a predetermined setof images of teeth of the patient at specified viewing angles relativeto the teeth using a user-detectable output from the mobiletelecommunications device, wherein the user-detectable output indicatesto the patient that the patient should start moving the camera of themobile telecommunications device relative to a mouth of the patient tothe right/left in order to capture the predetermined set of images ofteeth at the specified viewing angles; capturing the predetermined setof images of the teeth of the patient as the patient moves the camera ofthe mobile telecommunications device relative to the mouth of thepatient; displaying, during the capture of the predetermined set ofimages of the teeth of the patient, video from the camera of the mobiletelecommunications device to a screen of the mobile telecommunicationsdevice; uploading the captured predetermined set of images of the teethfor treatment progress evaluation; and providing feedback associatedwith the treatment progress evaluation to the patient.
 2. Thenon-transitory, computer-readable storage media of claim 1, wherein thecaptured predetermined set of images of the teeth are uploaded to aremote server.
 3. The non-transitory, computer-readable storage media ofclaim 1, wherein the patient is guided to capture the predetermined setof images in a predetermined sequence.
 4. The non-transitory,computer-readable storage media of claim 1, wherein the method furthercomprises recognizing and categorizing the teeth imaged by the camera ofthe mobile telecommunications device based on training.
 5. Thenon-transitory, computer-readable storage media of claim 1, wherein themethod further comprises automatic detection of teeth imaged by thecamera of the mobile telecommunications device by machine learning. 6.The non-transitory, computer-readable storage media of claim 1, whereinthe method further comprises detecting, using edge detection imageprocessing, the teeth imaged by the camera of the mobiletelecommunications device.
 7. The non-transitory, computer-readablestorage media of claim 1, wherein capturing the predetermined set ofimages of the teeth of the patient as the patient moves the camera ofthe mobile telecommunications device relative to the mouth of thepatient comprises detecting the teeth of the patient and automaticallyfocusing the camera of the mobile telecommunications device on thedetected teeth.
 8. The non-transitory, computer-readable storage mediaof claim 1, wherein capturing the predetermined set of images of theteeth of the patient as the patient moves the camera of the mobiletelecommunications device relative to the mouth of the patient comprisesautomatically adjusting an illumination provided by a light source ofthe mobile telecommunications device.
 9. The non-transitory,computer-readable storage media of claim 8, wherein the illuminationprovided by the light source of the mobile telecommunications device isadjusted to be a continuously on light.
 10. The non-transitory,computer-readable storage media of claim 1, wherein the specifiedviewing angles comprise an upper and lower jaw anterior viewing anglewith a closed bite, a right buccal viewing angle with the closed bite, aleft buccal viewing angle with the closed bite, an anterior viewingangle with an open bite, a right buccal viewing angle with the openbite, and a left buccal viewing angle with the open bite, and whereinthe method further comprises displaying a series of overlays to thescreen of the mobile telecommunications device, the series of overlayscomprising an overlay associated with the upper and lower anteriorviewing angle with the closed bite, and an overlay associated with theanterior viewing angle with the open bite.
 11. The non-transitory,computer-readable storage media of claim 10, wherein the series ofoverlays are displayed on the screen as solid.
 12. The non-transitory,computer-readable storage media of claim 10, wherein the series ofoverlays are displayed on the screen as transparent.
 13. Thenon-transitory, computer-readable storage media of claim 10, wherein theseries of overlays are displayed on the screen as an outline of teeth.14. The non-transitory computer-readable storage media of claim 1,wherein capturing the predetermined set of images of the teeth of thepatient as the patient moves the camera of the mobile telecommunicationsdevice relative to the mouth of the patient comprises detecting one ormore teeth imaged by the camera of the mobile telecommunications deviceand automatically taking an image when the detected one or more teethare in a desired orientation, size, and/or location on the screen. 15.The non-transitory computer-readable storage media of claim 14, whereincapturing the predetermined set of images of the teeth of the patient asthe patient moves the camera of the mobile telecommunications devicerelative to the mouth of the patient comprises using an overlay.
 16. Thenon-transitory computer-readable storage media of claim 14, whereincapturing the predetermined set of images of the teeth of the patient asthe patient moves the camera of the mobile telecommunications devicerelative to the mouth of the patient is performed without an overlay.17. The non-transitory, computer-readable storage media of claim 1,wherein, after capturing the predetermined set of images of the teeth,an image quality of the captured predetermined set of images is reviewedand an indication is displayed on the screen if image quality of one ormore of the captured predetermined set of images is below a threshold.18. The non-transitory, computer-readable storage media of claim 17,wherein the image quality comprises a focus.
 19. The non-transitory,computer-readable storage media of claim 17, wherein the image qualitycomprises a lighting.
 20. The non-transitory, computer-readable storagemedia of claim 17, wherein the image quality comprises a visibility ofall the teeth and wherein image recognition algorithms are used toreview of the predetermined set of images to assess the visibility ofall the teeth.
 21. The non-transitory, computer-readable storage mediaof claim 17, wherein the method further comprises showing a message onthe screen of the mobile telecommunications device to the patient toretake an alternative image if the image quality of one or more of thecaptured predetermined set of images is below the threshold.
 22. Thenon-transitory, computer-readable storage media of claim 1, wherein,after capturing the predetermined set of images of the teeth, thepredetermined set of images of the teeth are automatically labeled witha corresponding view that is detected.
 23. The non-transitory,computer-readable storage media of claim 1, wherein capturing thepredetermined set of images of the teeth of the patient as the patientmoves the camera of the mobile telecommunications device relative to themouth of the patient comprises using the camera in a continuous burstmode.
 24. The non-transitory, computer-readable storage media of claim23, wherein capturing the predetermined set of images of the teeth ofthe patient as the patient moves the camera of the mobiletelecommunications device relative to the mouth of the patient using thecamera in the continuous burst mode comprises automatically reviewingcaptured images.
 25. The non-transitory, computer-readable storage mediaof claim 24, wherein capturing the predetermined set of images of theteeth of the patient as the patient moves the camera of the mobiletelecommunications device relative to the mouth of the patient using thecamera in the continuous burst mode comprises automatically matchingviews to the specified viewing angles.
 26. The non-transitory,computer-readable storage media of claim 24, wherein automaticallyreviewing captured images comprises selecting only those images having asufficiently high quality for the predetermined set of images of theteeth.
 27. The non-transitory, computer-readable storage media of claim1, wherein guiding the patient to capture the predetermined set ofimages of teeth of the patient at specified viewing angles relative tothe teeth comprises displaying one or more arrows on the screen of themobile telecommunications device, the one or more arrows indicating thata user should move the mobile telecommunications device in a particulardirection.
 28. The non-transitory, computer-readable storage media ofclaim 1, wherein the method further comprises displaying a messageguiding the patient to take the predetermined set of images of the teethof the patient with a retractor on the screen of the mobiletelecommunications device.
 29. A non-transitory, computer-readablestorage media having executable instructions that are executed by aprocessor of a mobile telecommunications device to cause the mobiletelecommunications device having a camera to perform a method forcapturing dental images for analysis, the method comprising: guiding apatient to capture, with the camera of the mobile telecommunicationsdevice, a predetermined set of images of teeth of the patient atspecified viewing angles relative to the teeth using a user-detectableoutput from the mobile telecommunications device, the specified viewingangles comprising an upper and lower jaw anterior viewing angle with aclosed bite, a right buccal viewing angle with the closed bite, a leftbuccal viewing angle with the closed bite, an anterior viewing anglewith an open bite, a right buccal viewing angle with the open bite, anda left buccal viewing angle with the open bite, and wherein theuser-detectable output indicates to the patient that the patient shouldstart moving the camera of the mobile telecommunications device relativeto a mouth of the patient in order to capture the predetermined set ofimages of teeth at the specified viewing angles; capturing thepredetermined set of images of the teeth of the patient as the patientmoves the camera of the mobile telecommunications device relative to themouth of the patient by automatically adjusting an illumination providedby a light source of the mobile telecommunications device tocontinuously on; displaying, during the capture of the predetermined setof images of the teeth of the patient, (1) video from the camera of themobile telecommunications device to a screen of the mobiletelecommunications device, and (2) a series of overlays to the screen ofthe mobile telecommunications device, the series of overlays comprisingan overlay associated with the upper and lower anterior viewing anglewith the closed bite, and an overlay associated with the anteriorviewing angle with the open bite, wherein the screen is not facing thepatient during capture of the predetermined set of images; uploading thecaptured predetermined set of images of the teeth for treatment progressevaluation; and providing feedback associated with the treatmentprogress evaluation to the patient.
 30. A non-transitory,computer-readable storage media having executable instructions that areexecuted by a processor of a mobile telecommunications device to causethe mobile telecommunications device having a camera to perform a methodfor capturing dental images for analysis, the method comprising: guidinga patient to capture, with the camera of the mobile telecommunicationsdevice, a predetermined set of images of teeth of the patient atspecified viewing angles relative to the teeth using a user-detectablesound output from the mobile telecommunications device, wherein theuser-detectable sound output indicates to the patient that the patientshould start moving the camera of the mobile telecommunications devicerelative to a mouth of the patient in order to capture the predeterminedset of images of teeth at the specified viewing angles; capturing thepredetermined set of images of the teeth of the patient as the patientmoves the camera of the mobile telecommunications device relative to themouth of the patient; displaying, during the capture of thepredetermined set of images of the teeth of the patient, video from thecamera of the mobile telecommunications device to a screen of the mobiletelecommunications device so that from a perspective of the patient avideo is being taken, wherein the screen is not facing the patientduring capture of the predetermined set of images; after capturing thepredetermined set of images of the teeth, reviewing an image quality ofthe captured predetermined set of images, wherein the image qualitycomprises a visibility of all or a sub-set of the teeth and whereinimage recognition algorithms are used to review of the predetermined setof images to assess the visibility of all or a sub-set of the teeth; anddisplaying on the screen of the mobile telecommunications device aresult of the image quality review; uploading the captured predeterminedset of images of the teeth for treatment progress evaluation; providingfeedback associated with the treatment progress evaluation to thepatient; and wherein the teeth imaged by the camera of the mobiletelecommunications device are automatically recognized and categorizedbased on machine learning.